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ABSTRACT 



This evaluation study was conducted at the behest of the Aga 
Khan Foundation (AKF) and the Institute for Policy Reform (IPR) , working 
under contract with United States Agency for International Development. AKF ' s 
primary concern was to evaluate the effectiveness, cost-effectiveness, and 
the likelihood of sustainability of the School Improvement Programme (SIP) 
operating in Kisumu, Western Kenya. The SIP began in January 1990 with the 
intention to improve the quality of teaching and learning in primary schools 
in the Kisumu Municipality by promoting the adoption of child- centered 
teaching methods and developing strategies to institutionalize the process 
and outcomes of the effort. Data were gathered through standardized tests, 
observations, and interviews. Intervening variables were examined for their 
influence on the outcome of the tests. Overall, the likelihood that SIP'S 
impact will be sustained over time is highly questionable owing to a variety 
of factors including high rates of turnover in school staff and SIP staff, 
limited transportation, currently untrained SIP staff, triple the size of the 
target audience due to expanded municipal boundaries, overriding and 
non-child-centered influence of the KCPE (Kenyan Certificate of Primary 
Education) , and quality dissipation in training. Specific recommendations are 
offered for the program, training, and policy. The four chapters include: (1) 

"Background and Project Description"; (2) "Evaluation Study"; (3) "Findings"; 
and (4) "Discussion and Recommendations." (Contains 16 references.) (EH) 
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EXECUTIVE SUMMARY 



This evaluation study was conducted at the behest of the Aga Khan Foundation (AKF) and 
the Institute for Policy Reform (IPR), working under contract with USAID. AKF’s primary 
concern was to evaluate the effectiveness, cost-effectiveness and the likelihood of 
sustainability of the School Improvement Programme (SIP) operating in Kisumu, Western 
Kenya. IPR’s interest in SIP was as part of a larger collection of studies concerned with the 
effects of various school improvement interventions on enrolment and achievement in 
primary schools in Africa. 



The School Improvement Programme began in January 1990 with the intention to 
improve the quality of teaching and learning in primary schools in the Kisumu Municipality 
by promoting the adoption of child-centred teaching methods and developing strategies to 
institutionalize the process and outcomes of the effort. The strategy involved training teachers 
in a workshop setting, providing classroom-based technical assistance and support to teachers 
as they attempted to integrate what they learned into their day-to-day teaching behaviours, and 
providing instructional materials to support learning. 

SIP operated in two phases: Phase 1 from 1990 through 1992, working with teachers in 
15 schools in standards (grades) one to three; Phase 2 from early 1993 to August 1996, 
working with teachers in 27 schools in standards one to three and teachers in 13 of the same 
schools in standards four to six. This evaluation focused on the activities of Phase 2, although 
comparisons are made with teachers who were involved during Phase 1. 

The objectives for Phase 2 were to: extend the project to an additional 27 schools at the 
lower primary levels and to nine of these schools at the upper primary level; integrally involve 
head teachers in the training and improvement programme; continue making communities and 
parents aware of the programme activities and to motivate parents to contribute toward the 
cost of the teaching materials; improve the cost-effectiveness of the programme; and create a 
better understanding of child-centred learning. 
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Study Design 

Eleven treatment schools were selected to represent varying levels of SIP intervention. 
Four T1 schools were involved in the project during Phase I. Four T2 schools were involved 
in the early stages of Phase 2, having three-years for the training and TA to be internalized. 
Three T3 schools participated during January through May 1996. Four control schools were 
drawn from the Siaya district, northwest of Kisumu. Data was collected during June 1996. 

The research questions addressed in this study are: did the SIP intervention have a 

significant impact on increases in student achievement as compared with control schools; did 
SIP change the way that teachers teach to a more child-centred approach; was a change in 
teacher behaviour associated with increases in test scores; is the SIP model a cost-effective 
approach to improving teaching and learning; will the impact of SIP be sustained after 
external funds are removed? Tests were administered to approximately 650 pupils in each of 
standards 3 and 6; teachers were observed in these classes; and teachers, head teachers, 
parents, pupils, MEO employees, and project staff were interviewed. 

Findings 

Summary of Accomplishments: The SIP team worked with lower primary teachers in 
27 schools and upper primary teachers in 13 schools — four more than specified in the 
proposal; it provided instructional materials to over 100 classrooms and teachers and trained 
teachers in their use; quite late in the project it began working with head teachers; it met with 
the parents and the School Committee for each school to introduce them to the SIP model, to 
enjoin their participation and support and to encourage contributions toward the cost of 
materials; it established a close and productive relationship with the MEO and the Town 
Council resulting, in part, in replacement and training of many of the inspectors and staff of 
the Teacher Advisory Centres; it instilled an ethos of school improvement and attention to the 
concepts of child-centred teaching throughout the Municipality. 

Child-centred Teaching: Although SIP teachers exhibited substantially more child- 

centred teaching behaviours than did the control teachers, in most cases, their use of these 
behaviours was limited and shallow. The more important, yet difficult, behaviours such as 
asking questions that engage pupils in thinking analytically, having pupils be active learners 
working in small groups, encouraging children to express themselves and explore ideas or use 
language to communicate and understand, were rare in most classrooms observed. However, 
a few teachers exhibited exemplary teaching behaviours. 
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Student Learning: The analysis of test results was conducted in two stages: an analysis 
of variance (ANOVAs) to detect simple differences in scores across groups, and regression 
analyses to control for external factors that may contribute to results but are not attributable to 
the SIP treatment, such as family income and parents’ education. 

Standard 3: When family, teacher and school variables are not_considered, the SIP T1 
pupils performed significantly better than all other groups on all tests. The T2 pupils 
performed significantly better than the control pupils only in the English short-answer test. 
The T3 pupils generally performed worse than any other group on all tests. However, when 
non-treatment factors are controlled for, being in a SIP T2 school has a positive effect on test 
scores in English and math, as does the number of workshops a teacher attended. Teachers in 
T1 attended more workshops than any other group and so their pupils scores were increased 
by this factor rather than the more comprehensive SIP treatment which included the coaching 
of teachers and instructional materials. Teachers’ classroom management behaviours were 
also a significant influence on English scores, but child-centred teaching behaviours were not 
correlated with higher test scores. The most consistently influential variables were the 
number of years a teacher has been teaching and the number of workshops attended. When 
test scores were adjusted for family and teacher influence, pupils in T1 schools achieved 
scores across all tests administered for this study that were 16 percent greater than those of the 
control group, while pupils in T2 and T3 both achieved scores that were 5 percent greater than 
the control group, across all tests. 

Standard 6: When non-treatment variables are not considered, the SIP T1 test scores were 
significantly greater than the T3 and control schools in all tests except the math open-ended 
test. T2 schools’ performance was not greater than that of the control schools. However, 
when family variables are considered, SIP had no influence on any of the test scores for this 
standard. Although adjusted test scores at the standard 6 level were not significantly greater 
than scores of the control pupils, the SIP pupils at this level achieved a 13 percent increase in 
scores across all tests when compared with control pupils. 



The child-centred teaching behaviours promoted by SIP did not seem to have a positive 
influence on test scores at either grade level. Students in both treatment and control groups 
performed reasonably well on the short-answer and multiple-choice tests, suggesting that they 
are learning what is in the curriculum in the way that it is taught in the curriculum and 
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textbooks. However, scores were exceptionally poor on all open-ended tests for all groups - 
around 10%. Many pupils wrote nothing at all or simply rewrote the prompt. There is clear 
evidence that they are not learning how to read, write, or communicate in English or 
Kiswahili. Nor are they learning to apply simple mathematical concepts and skills to real-life 
types of problems. 

Parent Involvement and Views: SIP offered training to School Committees in SIP 

schools and encouraged Committees and individual parents to contribute more to the schools 
- significantly more than in control schools. Seventy percent of the SIP parents reported that 
their children seemed more confident and were more likely to speak freely since SIP 
involvement. Significantly more SIP head teachers reported an increase in the number of 
school visits by parents since SIP participation, compared with head teachers in control 

schools. 

Pupils’ Views: Almost all pupils in both class levels and in treatment and control groups, 
reported liking school, liking to leam, and having a teacher who likes them. 

Teachers’ and Head Teachers’ Views: Interviews with teachers and head teachers 

revealed very positive attitudes toward the SIP project and Programme Officers. Teachers 
seemed to see and believe in the value of child-centred teaching strategies, but were reluctant 
to adopt fully these behaviours because they felt a pressure to cover the curriculum and ensure 
that pupils were prepared to take and succeed in the national primary certification 

examinations. 

Materials: SIP significantly enhanced the classrooms in which it worked with 

instructionally-useful materials, and wisely established a mechanism to ensure that all 
teachers within the Municipality would have access to similar materials. However, the 
examinations developed by SIP are not supportive of child-centred teaching and should be 
discontinued until revised. SIP’s research and monitoring efforts were exemplary and useful 
to the project. 

Staff Training: SIP staff were not trained to a sufficient level of mastery to do their jobs. 
They reported that they had not received any guidelines or training in what to do when they 
visited a school, nor were they taught how to do a demonstration lesson or give feedback to 
teachers. They indicated that they would have preferred training and coaching that gave them 
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time to reflect in groups, to read and discuss original source documents, and to have the 
expert(s) observe them in the classrooms as they worked with teachers. 

Shortcomings: 1) SIP waited until very late to work with head teachers; 2) they did not 
focus on the whole school; 3) SIP staff were not adequately trained; 4) they did not appear to 
bring project teachers to a sufficient level of skill mastery in using child-centred techniques or 
in teaching children to read and write; and 5) there was a high rate of staff turnover. 

Constraints: 1) A high rate of school and MEO staff transfers; 2) a high rate of school 

closings for multiple examinations, athletic events and music festivals; 3) teacher pressure to 
cover a very full curriculum and to ensure that pupils are successful in the KCPE examination 
which is factually oriented and not particularly child centred; 4) a national curriculum that 
emphasizes grammar to the neglect of reading; 5) a high rate of teacher absences, in part due 
to the high rate of AIDS-related illnesses and deaths in Kisumu but also due to lax supervision 
on the part of the head teachers and the MEO; 6) pressure to teach in English beginning in the 
very early grades. 

Costs: SIP significantly reduced the cost-per-teacher (educator) -trained by 61% during 
Phase 2. However, the cost of increasing a pupil’s test score by one percent ranged from 
about KSH 300/ to 800/, or $US 5.00 to $14.00 per pupil. Relative to the standard Kenyan 
1993 per-pupil-expenditure of Ksh 2131/, SIP spent about 15% more for each additional 
percent increase in test scores for pupils in groups Tl, T3 and the standard 6 pupils, but 36% 
more for pupils in T2 schools. These figures are fairly high and probably not viable costs for 
a government agency with limited resources to incur, but they can serve as a benchmark for 
comparing with other approaches SIP may try to increase learning. 

Sustainability: Overall, the likelihood that SIP s impact will be sustained over time is 
highly questionable owing to a variety of factors including high rates of turnover in school 
staff and SIP staff, limited transportation, currently untrained SIP staff, triple the size of the 
target audience due to expanded municipal boundaries, overriding and non-child-centred 
influence of the KCPE, and quality dissipation in training. 
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Recommendations 
Programme - r •' 

Focus on supporting the development of the school as a whole rather than only selected 
teachers within the school. 

Develop the head teacher as the instructional leader — continue working with School 
Coordinators. 

Emphasize the teaching of reading, writing and thinking, and develop teachers questioning 
strategies. 

Encourage the MEO to assume more responsibility over school accountability issues such as 
attendance and school closures and to reduce the burden of multiple examinations. 

Work with a low-cost school to use as a model to show that teachers can use child-centred 
practices, cover the curriculum, and help pupils achieve high test scores. 

Develop, administer and report practice examinations that are consistent with the child- 

centred approach promoted by SIP . 

Establish an effective-schools peer review process using research-based indicators. 

Training 

Establish a procedure and measures to ensure that all programme staff are trained to mastery 
in the content and skills needed to do their jobs effectively. 

Conduct a study of the extent of dissipation in the quality of training and technical assistance 
as it moves from expert trainer, to programme officers, to school co-ordinators, and to 

teachers. 

Consider establishing a centralized training academy for staff developers, district educational 
managers, and head teachers. 

Consider using technology, such as CD-Rom, to provide training that will retain the quality of 
the “ expert” trainer and can be used across programmes. 

Consider developing training materials centrally for all AKF education projects. 

Policy 

Conduct a study of teacher training colleges to assess the quality of teaching and supervision 
of these institutions. 

Consider ways to influence changes in national curriculum and examinations to make more 
compatible with child-centred instructional approaches and to increase the focus on teaching 
reading and writing. 
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CHAPTER 1 

- , BACKGROUND AND PROJECT DESCRIPTION 



Purpose of the Study 

This evaluation study was conducted at the behest of two sponsors, the Aga Khan 
Foundation (AKF) and the Institute for Policy Reform (IPR). AKF’s primary concern was to 
evaluate the effectiveness, cost-effectiveness and the likelihood of sustainability of the second 
phase of the Kisumu School Improvement Programme (SIP) operating in Western Kenya. It 
was also concerned that the problems and recommendations identified in the evaluation of the 
first phase of the project had been addressed. IPR, working under contract with the United 
States Agency for International Development (USAID/ART) was interested in Kisumu SIP is 
as part of a larger collection of studies concerned with research that would assist African 
policymakers in selecting budget allocations for education that would maximize the 
developmental benefits of educational expenditures. The collection of studies, conducted in 
several African countries, all intended to clarify the linkages between the inputs to, and 
outcomes of, the education system. In four of the studies, the purpose was to identify the 
effects of various school improvement interventions on enrolment and achievement. Toward 
this end, IPR identified several existing projects that were intending to conduct end-of-term 
evaluations, and entered into agreements to jointly support these studies. The Kisumu School 
Improvement Programme was one such project. 

AKF’s goals for this study were to evaluate the extent to which the project was 
implemented as intended, whether it made an impact on changing teachers’ behaviours toward 
a more child-centred approach, whether the change in behaviours resulted in any improvement 
in student learning, whether the cost-per-teacher trained was reduced, and to estimate the 
likelihood that the project would be sustained when handed over to the local government. To 
measure the extent to which these goals were achieved, tests were administered to pupils in 
standards (grades) 3 and 6 and data also were collected through classroom observations, 
interviews with teachers, head teachers, entire school staff, Municipal staff, parents, 
community members, pupils and project staff. 

The Aga Khan Foundation and Its Strategy for school Improvement 

The Aga Khan Foundation (AKF) is a private, non-profit organization founded by His 
Highness the Aga Khan, the 49th hereditary Imam of the Ismaili Muslims. Established in 
1967 as a non-denominational philanthropic institution, 1 the Foundation addresses the needs of 
beneficiaries without regard to race, religion, gender or political persuasion. It is part of a 
larger group of social, economic, and cultural development institutions generally referred to as 
the Aga Khan Development Network with origins reaching back to the 1890s, and includes 
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the Aga Khan Health and Education Services, which operate some 500 health and education 
facilities and programmes, primarily in the developing world. The Aga Khan Foundation is 
intended to support innovative development programmes that help foster lasting solutions to 
enduring social development problems, particularly in health, education, and improvement in 
the quality of the lives of rural people in developing countries. Two priorities of the 
Foundation are to create institutional structures that encourage participation and increase the 
capacities of people to determine their own development priorities, and to enable communities 
to solve human problems and alleviate suffering through private sector, non-profit initiatives. 
More recently, the International Strategy of the AKF is to promote co-operation between 
government, business and private voluntary agencies to create a more enabling environment 
for progress.” 

In education, the Foundation supports programmes in early childhood care and 
development, and school improvement, including teacher training, developing and producing 
affordable teaching materials, and improving the management of schools. Since the early 
1980s, the Foundation has promoted a series of school improvement efforts in India, Pakistan, 
Kenya, Tanzania, and more recently, Uganda. Two approaches have been used: 1) projects 
focused on individual schools and aimed at changing the ethos of school staff and managers 
toward teaching and learning; 2) projects that have focused on influencing a larger number of 
schools through the use of a team of master trainers who provide in-service training on site at 
schools. 



The intent in both approaches is to increase the extent of activity-based learning 
through greater student participation in activities and group discussions. Children are 
encouraged to question more and teachers are encouraged to have pupils work in small groups 
and to make and use materials to support learning. These instructional strategies are referred 
to by AKF as a child-centred approach to teaching and learning and are central to the School 
Improvement Programme implemented in Kisumu, Kenya and evaluated in this study. 

Background on the Kisumu School Improvement Program 

The School Improvement Programme - Aga Khan (SIPAK) began in January 1990. 
The intent was to improve the quality of teaching and learning in primary schools in the 
Kisumu Municipality by promoting the adoption of child-centred teaching methods and 
developing strategies to institutionalize the process and outcomes of the effort. The strategy 
involved training teachers in a workshop setting, providing classroom-based technical 
assistance and support to teachers as they attempted to integrate what they learned into their 
day-to-day teaching behaviours, and providing instructional materials to support learning. 
SIP’s emphasis on child-centred methods was consistent with the stated goals advocated by 
the Kenya Ministry of Education and the pre-service approach of the Kenyan teacher training 
colleges. 
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The project began in response to declining academic standards in the Kisumu Municipality as 
reflected in the primary leaving examination scores (Kenya Certificate of Primary Education - 
KCPE). In 1988, the Municipal Education Office (MEO) in Kisumu commissioned a survey 
of the primary schools in the Municipality to try to identify the reasons for the decline. The 
results of the survey indicated several contributors: instruction was dominated by teachers 
who placed an emphasis on choral responses and rote memorization, and offered students little 
opportunity to discuss or interact; group work was infrequent and there were no opportunities 
for pupils to engage in discovery or problem solving; there was a shortage of textbooks and 
other materials used in the classrooms; the classroom culture was dominated by tests and 
examinations and completion of the national curriculum; physical facilities such as classrooms 
and desks were in short supply; teacher morale was quite low; pupil-teacher ratios were high, 
school management was inadequate and ineffective; parents and community had little 
commitment to raise the funds needed to operate the schools; parents and other local leaders 
were not involved in schooling nor did they understand modem educational methods, there 
was poor understanding of the newly-introduced 8-4-4 curriculum. 

In addition, when the Aga Khan Education Service (AKES) reviewed the teaching 
practices in their nursery and primary schools in Kisumu, they found that the child-centred 
teaching methods employed in the nursery school were not being used in the primary school. 

Shortly after these studies were completed, the Kisumu Municipality and AKES 
joined forces to seek funds to support a school improvement effort. Support was provided by 
several donor agencies, including AKF, the Canadian International Development Agency 
(CIDA), the UK Overseas Development Administration, and Comic Relief (UK). 

The SIP project was designed in accordance with two themes of the AKF’s Strategy 
on School Improvement: 1) involvement of teachers in developing curriculum materials for 
their own use in the classroom, and 2) training and coaching teachers in new instructional 
skills and approaches within the context of their own classrooms. 

Other areas of emphasis emerged during the project: staff training for the extended 
school community including parents, inspectors and TAC tutors; and the establishment o a 
close working relationship with the inspectors and tutors. (TACs are Teacher visory 
Centres established by the Kenyan Ministry of Education and managed by local Municipal 
Education Offices (MEOs). These Centres are dispersed throughout the country and were 
established to provide a resource for teachers in developing instructional materials; co- 
ordinating and providing in-service training; co-ordinating the work of subject panels; 
providing professional guidance and counselling to teachers; and serving as a link between 
schools and other educational institutions such as the Kenya Institute of Education, the Kenya 
National Examinations Council, the Teachers’ Service Commission, the inspectorate, and 
colleges and universities. The head of each TAC is referred to as a TAC tutor. In Kisumu the 
MEO recently established one TAC in each of the nine municipal zones and SIP assisted oy 
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providing start-up funds for materials such as textbooks, teacher reference books, copiers, 
typewriters and office supplies.) 

SIP was governed by a Project Implementation Committee (PIC), chaired by the 
Municipal Education Officer (MEO), with the local Aga Khan Education Service (AKES) 
Chair serving as the Vice-chair. Other members included the SIP Project Director, the 
Executive Officer of AKES-Kenya, a head teacher, and a representative of a local teachers 
college. No classroom teachers served on the Committee. This structure was designed to 
establish a precedence of ownership by the Kisumu Municipality. The function of the PIC 
was to oversee the managerial and financial dimensions of the project and to serve as advisors 
to the Project Director. 

SIP - Phase 1 

SIP has been in operation for six years, from January 1990 through August 1996. 
Phase I lasted for three years from 1990 through the end of 1992, and Phase 2 from early 1993 
to August 1996, at which time the project became a part of the Kisumu Municipal Education 
Office. There were several differences in the project during the two phases. At the beginning 
of Phase I, three model schools were selected, beginning with the Aga Khan Primary School, 
a high-cost school, and shortly thereafter two other schools were included to represent peri- 
urban and urban low-cost schools. 

The Kenyan Education System currently has two types of school designations - public 
and private. Public schools are those run by parents in the form of a school committee. The 
parents provide the school facilities while the central government pays teacher and head 
teacher salaries and provides inspection and supervision services. Within the public-school 
category, there are both high- and low-cost schools. In Kisumu, high-cost schools are public 
schools which were once Asian or European schools with high-quality facilities. These 
schools pay for the services of groundsmen, cleaners, and clerks, thereby requiring that 
parents pay a higher fee to maintain the facilities. Often, they also employ an accounts clerk 
to maintain school accounts, thereby freeing the head teachers’ time for other responsibilities. 
Low-cost schools are primarily community schools that levy small amounts of money for 
services and school development. (More detail is provided on this topic in the section on study 
sample.) 
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A factor in the selection of one of the first SIP schools was that the lower-primary 
children were learning in semi-permanent buildings with mud walls. The other school had 
fairly high-quality facilities but a fee structure typical of the low-cost schools, thereby more 
likely to accommodate more low-income parents than the Aga Khan Primary School (AKPS). 
Other selection criteria included a positive attitude toward the project goals, proximity and 
accessibility of the school to the SEP project headquarters (based on the grounds of the 
AKPS), community support and involvement, need for improvement, and school security to 
protect the materials that would be introduced to the classrooms through the project. 

Initially, the training and technical assistance (coaching) was offered to teachers in 
classes 1 to 4, but after the first year class 4 was dropped since at that grade the teachers teach 
one or two subjects to different classes of pupils rather than teach all subjects to one class. 
Each of the three charter schools was used as a “model” school and, after a year, three more 
schools were added. By the end of Phase 1, 15 schools had received SIP training and 
technical assistance. 

One of the distinguishing characteristics of Phase I is the use of master teachers. 
Eight teachers per school (a total of 24) were trained during the first year of the project along 
with the project staff, who are referred to as Program Officers (POs). After receiving training, 
the 24 teachers became master teachers and continued with their classroom teaching 
responsibilities, but also were used as project trainers along with the POs. The master 
teachers were paid KSH 1000 per month in addition to their teaching salaries. Each model 
school was paired with a new school and one project staff member and one master teacher 
was assigned to each stream, or grade, within that school. This team of six (two for each of 
the three streams) would work intensively with their assigned teacher for a period of six 
months, providing training and in-classroom coaching. 

The project staff member or master teacher would typically conduct one or more 
demonstration lessons in the target teacher’s classroom while the teacher observed. This was 
followed by a discussion and analysis of the lesson, and a joint planning session for a lesson 
the teacher could do the next day that integrated some of the new behaviours demonstrated by 
the project staff or master teacher. 
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Phase I Evaluation: At the end of 1992, an evaluation of Phase 1 activities was 
conducted and' the findings of that study influenced the design of the second phase of SIP 
activities. In that evaluation, data was collected from four schools, one each representing 
various levels of SIP intervention and one control school. The report concluded that there was 
evidence of substantial changes in classroom practice but that teachers’ use of 
individualization and group work was limited. They noted that the clinical approach to 
developing teachers’ competence was an effective model. However, it expressed concern that 
the project was not encouraging sufficient depth in teachers’ thinking about the teaching 
methods encouraged by SIP. There was a concern that teachers may adopt certain practices to 
please the trainer rather than because the teacher was convinced of the effectiveness of the 
practices. Moreover, if teachers’ understanding of the various practices were not sufficiently 
deep, or suffered from misconceptions, then they would not know when to apply one practice 
over another — for example, when to engage students in group work versus teaching to the 

whole class. 

There was also a concern that the project had not provided sufficient attention to the 
school as a whole and to the role of the head teacher in school improvement. Specifically, the 
evaluators indicated that the project had not offered training to head teachers in how to 
support and monitor teachers in a way that would ensure sustained improvements. 

Results showed that student learning had “not declined as a result of the intervention” 
and the authors reported some evidence of gains in language skills. Classroom environments 
were viewed as more stimulating and pupils seemed more self-confident and better able to 
take responsibility for their own learning. 

Further recommendations for Phase 2 included: increase dissemination of the 

project’s model by having the Project Director write descriptive papers about their work and 
have AKF share its work with other donors through international conferences; make clear to 
the beneficiaries that the project’s intent is to “prime the pump” and that improvements made 
by the project should be sustained within existing local budgets; clarify project duration in 
relation to staff employment in order to ensure staff productivity throughout the entire project 
cycle; nurture skills development of project staff; require that the project proposal be more 
precise in describing intended outcomes; increase attention to systematic self-evaluation. 
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Many of these recommendations were incorporated into the second phase of SIP: the 
project began training head teachers and upper primary teachers, although this began quite late 
in Phase 2; several of the project staff were enrolled in a degree programme with a British 
university and several experts were brought in to conduct training for both project staff and 
beneficiaries; information on the project model was disseminated at an international 
conference and with a new AKF-sponsored SIP project starting up in Mombasa; and the 
project began engaging in self-evaluation through a number of studies of project impact. 
However, SEP never did address the school as a whole, and although a great deal of effort was 
put into addressing the future employment of project staff, this issue was still unresolved at 
the time of the Phase 2 evaluation. Some of the staff were promoted to zonal inspectors by the 
MEO, one became the new SIP Project Director, one retired from service, and one was hired 
by a local teacher training college. Only one continued on as a Programme Officer for SIP 
work. 

SIP - Phase 2 (September 1993 - August 1996) 

New leadership was brought in to guide SIP during the second phase of the project, 
and under this leadership, SIP became much more expansive, reaching beyond the objectives 
stated in the project proposal, extending beyond the lower primary classrooms into the upper 
grades, out to parents, community, MEO staff (including inspectors, TAC tutors), head 
teachers, and local universities and teacher training colleges. During the three years of Phase 
I, SEP staff worked in 15 schools, but during Phase 2, 27 new schools were addressed, with 
occasional visits to the original 15 for a total of 42 schools served. In addition, teachers and 
head teachers from a number of other schools attended open SIP workshops. The British 
Overseas Development Agency (ODA), the Canadian International Development Agency 
(CIDA) and AKF supported the second phase of this school improvement project. 

A major element of the restructured SIP was to more aggressively collaborate with the 
staff of the Municipal Education Office, particularly the Municipal Education Officer (MEO), 
the Deputy MEO, the inspectors, and the TAC tutors. Although this was specified as an 
element of the first phase of SEP activities, this outreach was not realized and sometimes 
resulted in an incompatibility between the SIP and MEO activities. For example, the 
inspectors were using forms and guidelines to evaluate schools that were inconsistent with the 
SIP objectives, causing confusion and frustration for SEP-trained teachers and head teachers. 
To alleviate this conflict, Phase 2 staff invited and encouraged participation of the inspectors 
and TAC tutors in SIP workshops and planning activities, and, with the co-operation of the 
MEO, redesigned the inspector’s school review and reporting form to reflect SIP goals and 
objectives. Collaborative relationships also were nurtured between the SIP staff and school 
staff. SIP staff worked closely with teachers in their classrooms, using discussions, 
observations, and critiques of lessons to encourage deep understanding of the desired teaching 
behaviours rather than a mechanical adaptation. Teachers were encouraged to negotiate, 
challenge, and accept or reject ideas proffered by SIP trainers. 
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While-Phase 1 was governed by an oversight committee that addressed managerial 
and financial dimensions of the project, it was felt that the educational decisions of SIP should 
reflect local views. An education committee was formed and referred to as the SIPAK 
Standing Committee. This Committee, which was chaired by the Project Director, met 
monthly and included representatives of the zonal inspectors, the TAC tutors, a head teacher, 
teacher, and an external member who was a parent and lecturer at a local teacher training 
college. 



School Co-ordinators: Instead of master teachers, the project introduced the role of 
School Co-ordinators. Co-ordinators were teachers in the target schools and were selected by 
the school’s head teacher as a potential leader in the school. Instead of working at other 
schools as the master teachers did, the Co-ordinators served as resident experts working 
alongside their colleagues in their own classrooms. They received additional, specialized 
training to help them assist teachers in implementing SEP activities once the SEP team moved 
on to other schools. The School Co-ordinators did not receive the additional stipend offered 
to the Master Teachers during Phase I, which was identified as a source of some resentment 
during both evaluations. One problem with the use of master teachers was that the schools 
from which they came were required to hire temporary staff to replace them. This was not a 
popular solution among school administrators, parents, or pupils. 

Another difference between Phases 1 and 2 is that in Phase 1 the SIP team members 
would replace a regular classroom teacher during the school visits, but in Phase 2, the SIP PO 
would demonstrate a lesson or team teach with the classroom teacher in his or her classroom. 
These were followed by discussions and collaborative preparation of activities for the next 

day’s lessons. 

During Phase 2, the project focused on teachers at both the lower and upper primary 
levels, including classes 1 through 6. Each academic term, four new schools were selected for 
focused attention. During that term (4 months), the SIP POs would work with the teachers on 
a daily basis, but at the end of the term, the attention was reduced substantially, as four new 
schools became the focus of service. The earlier schools would receive intermittent assistance 
and support depending on their needs and requests and time available. 
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Phase 2 schools were selected based on several factors, including: positive head 
teacher and teacher attitudes towards the program, secure classrooms to house learning 
materials, adequate desks, and zonal representation. Schools that did not have lockable doors 
and windows or sufficient desks were required to obtain them as a requirement for 
participation. This was intended to secure the classrooms against vandalism and theft of 
teaching materials and resources provided by the project. In fact, during the preliminary visit 
to Kisumu to prepare for the evaluation study, one school was visited that was in the process 
of building desks and installing windows and door security. The head teacher was very 
enthusiastic about the prospect of SIP participation and reported that it served as a motivator 
for the school staff and parents and helped to generate the funds needed for this hardware. 

Assessment: Several assessments were conducted by SIP staff to identify students 
specific learning needs. In early 1994, SIP administered a series of diagnostic tests in reading, 
writing, and comprehension to standards 1, 2, and 3 pupils respectively in several schools. 
The results revealed poor reading comprehension, and SIP designed and conducted workshops 
to address these deficiencies. Another assessment of the reading ability of 406 standard 5 
pupils was conducted on an individual pupil basis. The results showed that about 65% of the 
pupils were rated as good readers on measures of word recognition and reading of sentences, 
while 26% were rated as non-readers. In early 1995, another series of tests was developed in 
reading and administered to approximately 2,500 pupils in each of classes 1-7. The results for 
standard 1, for example, showed that 61% of the children could not recognize most of the 

letters of the alphabet. 

A far more comprehensive effort was begun in 1995 to develop practice examinations. 
This activity was based on the observation that schools within the Municipality were 
purchasing practice exams from other districts to help prepare pupils for the Kenya Certificate 
of Primary Education (KCPE) Examinations administered by the Kenya National 
Examinations Council (KNEC) at the end of primary school. The results on the KCPE 
examination are used to determine which pupils continue on to secondary school and so attract 
a great deal of attention of parents, teachers and pupils. In fact schools are expected to do all 
that they can to prepare their pupils to do well on these examinations and each school and 
district/Municipality are ranked on a national basis. (Kisumu’s ten-year rankings are provided 
in a later section of this report.) The taking of practice examinations begins in standard 1 and 
the end-of-year examination marks are combined with the frequent testing that occurs 
throughout the year, resulting in a final mark for each pupil in each of the subjects tested. 
The average of the marks in all subjects is used to rank the pupils in a class. 
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SIP staff observed that the practice exams purchased from outside the Municipality 
did not reflect- what was being taught in the schools in Kisumu, particularly in the areas of 
geography and agriculture. SIP initiated the development and administration of a set of local 
examinations to provide a standardized measure of programme activities, to improve the skill 
level of teachers in test development, and to have tests that were more appropriate to the local 
context. They developed a table of specifications similar in mode and format to the KCPE 
and engaged teachers in project and non-project schools to write test items based on the 
specifications. These items were then used by four subject panels to moderate the items and 
to form tests, which were then sent to the KNEC for evaluation. These practice exams were 
administered to 12 standard 3 classes in 1994 and used in 26 schools in standards 1-6 in 1995 
in science, mathematics, English and Kiswahili. 

Training: Throughout the three years of Phase 2 a variety of Kenyan experts were 
enlisted to conduct training for Kisumu educators. Expertise was drawn from the local teacher 
training colleges, the Kenya Institute of Education, and the Kenya National Examinations 
Council, although many of the workshop sessions were conducted by the Project Director. 
Training for teachers was intended to provide the practical hands-on experiences that were not 
common during their pre-service training and to reinforce many of the topics that were 
covered in teacher training. Workshop topics included questioning, libraries, reading, 
Kiswahili, curriculum interpretation, art and crafts, and learning centres. Many of these 
workshops served two purposes: 1) to provide additional training for the SIP staff, and 2) to 
train the range of other educators within the Municipality, including teachers, head teachers, 
School Co-ordinators, inspectors, and TAC tutors. After receiving training from local experts, 
the SIP project staff conducted subsequent workshops throughout the Municipality. The 
workshops were then followed up by classroom coaching that included teacher observations, 
discussions and demonstrations by the POs. Inspectors and TAC tutors were offered special 
workshops on their role as staff developers and advisors. School Co-ordinators were offered 
courses in syllabus interpretation, writing schemes of work, leadership, classroom 
management and organization, and peer coaching. School Committees and PTAs were 
offered training in school management, bookkeeping, budgeting, the relationship between the 
Committee and the school’s administration, and the role of the School Committee. 

As of May 1996, SIP reported having offered 47 workshops during Phase 2 with an 
overall attendance of 1,742. Additional workshops were being conducted during the 
evaluation. Two workshops of two days each were conducted in reading beginning in 1995 
after the staff identified reading to be a problem in the schools. It is important to note that 
attendance at any of these workshops was voluntary and therefore the number and type of 
workshops attended by teachers (or head teachers) within a school varied considerably. There 
was no guarantee that all the lower primary teachers within or across SIP schools had attended 
the same number or type of training. 
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In early 1994, SIP began offering training and technical assistance to teachers at the 
upper primary' level (standards 4-6). At the time of the evaluation, teachers in 13 SIP schools 
had received training and TA. Workshop topics included questioning strategies, pupil 
assessment, and interpretation and use of examination results. 

Training and technical assistance was not typically offered to an entire school staff, 
although if a staff requested assistance or the PO felt there was a specific school-wide need, 
school-based sessions were offered. In this regard, the title of the project is a misnomer. The 
focus of improvement was much more on the individual teacher rather than on the school as a 
whole. Teacher Development Project would have been more apt as a title. 

From late 1994, a greater focus was given to the training of head teachers. Workshops 
were offered in teacher evaluation, school management and organization, bookkeeping and 
funds management, leadership, approaches to teaching and learning, qualities of an effective 
school, and syllabus interpretation. The inspectors and TAC tutors were encouraged to 
provide follow-up support to head teachers. 



Parents and communities are influential to a school s success, particularly since many 
of the funds that support a school must come from them. Parents are expected to provide 
uniforms, textbooks and school supplies for their own children and in some schools are 
required to pay admission fees. In addition, schools often hold fund-raising events called 
harambees to support development or capital activities such as building latrines and additional 
classrooms. Before bringing a school into SIP, the SIP staff met with each school’s 
PTA/School Committee to explain the purpose of SIP and to encourage financial and 
academic support. In addition, they facilitated parents’ visits to other SIP schools to see the 

impact of SIP services. 

Links with Higher Education: In early 1995, discussions were held with faculty of 
education at a local university to explore various avenues of collaboration. The faculty were 
invited to encourage graduate education students to submit research proposals for possible 
AKF funding. The research topics were to relate to school improvement and the research 
would be supervised by Maseno University education faculty. At the time of the evaluation 
data collection, six proposals had been submitted. Research topics included: a study of the 
effects of the language of instruction on learning and teaching in lower primary classes; a 
study of the comprehensibility of mathematics topics; and a study of the effectiveness o 
teachers’ centres in the professional development of primary school teachers. 
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SIP staff and AKF also initiated discussions with Maseno University education faculty 
about the possibility of establishing a degree program in primary education. The purpose of 
the programme would be to train primary school teachers, tutors (faculty) for the primary 
teacher training colleges, curriculum developers, education supervisors, inspectors, 
administrators and TAC tutors. At present, the training of primary teachers is done at the 
primary teacher training colleges and the teaching of secondary school teachers is done at the 
universities. The plans also include establishing a centre for research in various areas of 
primary education. These discussions had not been resolved at the time of the evaluation. 

Promoting Sustainability: Steps were taken to ensure that SIP efforts are sustained. 
Increasingly, responsibilities for SIP activities involved or were handed over to the MEO, 
including planning workshop sessions, conducting workshops, replacing, upgrading, and 
training inspectors and TAC tutors, and creating a revised reporting format for inspectors to 
use that is more consistent with SIP instructional strategies. 

In addition, a system was established to place the procurement of instructional 
materials in the hands of the schools and the MOE’s office. A central feature of the SIP 
programme has been to provide SIP teachers with classroom materials, generally providing 
the raw materials and working with teachers to show them how to construct and use the 
materials as learning aids and learning centres. Beginning in 1995, each parent has been 
asked to contribute KSH 50/ - ($1US) per child per year for the purchase of consumable 
materials such as manila sheets, newsprint, glue and other items that the SIP project has been 
supplying. This money will go into a central Municipal fund to allow the purchase of 
materials in bulk at reduced costs, which will then be distributed to the schools based on the 
amount of funds submitted. 
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CHAPTER 2 
EVALUATION STUDY 



Described in this section are the study questions, study design, data collection 
procedures, a description of the treatment and control districts, the sample selection, and the 
data collection instruments, including achievement tests. Detailed information regarding the 
tests is in Appendix A and marking (or scoring) guidelines for the tests are in Appendix B. 

Study Questions 

The purpose of this evaluation study was to assess how well the SIP programme has 
operated, whether and how its operations could be improved and its costs lowered, and what 
needs to be done to maximize the chances that the improvements in teaching practices can be 
sustained once the operation of the training and supervision system is turned over to the 
Municipal authorities. The more general evaluation questions are: 

■ What difference has SIP made in student achievement and participation, independent of other 
factors that might explain these results. 

■ What explains the differences found? Which particular aspects of the programme seem to be 
the most important? 

■ What were the costs involved in achieving these results and which levels of intervention are 
most cost-effective? 

■ How sustainable are these results likely to be once the programme is turned over to the 
Municipality? 

■ What recommendations can be made to improve future operations: to enhance the skills and 
knowledge base of teachers, the effectiveness of head teachers, and the commitment and 
involvement of parents, and to reduce the costs of accomplishing these goals in the future? 



AKF had several more specific concerns regarding project implementation which are also 
addressed in this report. 

Study Design 

No baseline data was collected at the beginning of the SIP project so a pre-post 
treatment design was not possible. However, an interim evaluation was conducted at the end 
of Phase 1 and could have served as a baseline for Phase 2, but the data for that study was 
only collected on Phase 1 schools which were not “ treated” during Phase 2. Several other 
problems precluded the use of that sample and instruments for this study. For example, the 
English test was not available and the mathematics test appeared to be much easier than the 
skills found in the standard 3 curriculum. The tests developed for this study were more 
comprehensive and representative of the Kenya curriculum. In addition, the earlier study did 
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not address the upper-primary level since SIP had not been working at that level. Moreover, 
only three SIP schools were used for that study, with one being a school from which the 
master teachers had come — a practice no longer used. Any findings generated from that study 
could be attributed to the idiosyncrasies of that particular school and would not necessarily 
generalize to the population of SIP schools. The sample size for this study was much larger - 
1 1 SIP and 4 control schools. In addition, since the SIP treatment was focused on the teacher 
rather than the school, the teacher and his or her pupils were the unit of analysis. There have 
been a very large number of transfers and even deaths in the Kisumu municipality and the 
likelihood of finding the same teacher at the same school was small. Moreover, the 
comparison only would have applied to the Phase 1 sub-sample and not to the Phase 2 sample. 
However, many of the items used on the teacher questionnaire and classroom observation in 
the Phase 1 evaluation were included in those developed for this evaluation. 

Given that a pre-post design was not the best option, a treatment - control group 
design was used, with three levels of SIP intervention forming three treatment groups and a 
control group drawn from a nearby district, since SIP staff reported that all schools within 
Kisumu had been impacted by SIP in one way or another. In addition, extensive qualitative 
data was collected from teachers, head teachers, parents and pupils. 

Procedures 

During the last week of May 1996, local data collection staff were hired and trained 
and the instruments were pilot tested and revised. Data collection began June 3 and continued 
through June 20, with each of the 17 sample schools visited by a team of two senior 
researchers from about 7:45 a.m. until mid-aftemoon. (Only 15 of these schools were used in 
the final analysis. Two schools were included that were not officially SIP schools but had 
received some SIP impact: several of the teachers at one school had attended many SIP 
workshops, and the head teacher and a teacher at another school had been “ treated” while 
working at another SIP school. However, it was ultimately decided that generalizations based 
on a single school would not be meaningful.) Although all SIP schools had been notified that 
an evaluation team would be collecting data during the month of June, none knew if they were 
in the sample, nor when the site visit would be conducted, to ensure that school staff 
behaviour was as typical as possible. 
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The day began with a brief meeting with the head teacher to indicate the purpose of 
our visit and to schedule the day’s activities. In each school, one standard 3 and one standard 
6 teacher was observed, typically for two class periods - about 75 minutes each. We specified 
that we preferred to observe the teachers teaching English, maths, or science. All of the 
standard 3 (treatment) teachers were SIP trained with the exception of two schools where the 
standard 3 teacher had been transferred recently. In these schools, we observed a standard 2 
SIP-trained teacher, but tested the pupils of the former standard 3 SIP teacher. Only eight of 
the standard 6 upper primary teachers in the sample were SIP trained since SIP only began 
training teachers at the upper primary level in 1994. 

During these visits, each of the teachers observed and the head teacher was 
interviewed by one of the evaluators. Initially, the teachers were given a questionnaire to 
complete, but we began to see that they had difficulty responding to some of the questions, so 
eventually incorporated all questions into a single, interview instrument. Generally the 
questions seemed to be difficult because of the way they were structured for coding purposes. 
The head teachers seemed more able to complete the questionnaires on their own, so two 
separate instruments were maintained. During the lunch period the entire school staff was 
interviewed in focus group sessions. 

We asked if it was possible to meet that day with one or more representatives of the 
School Committee/PTA, but often found that we needed to schedule a meeting at a later time, 
in which case, one of the research team members returned for the interview. 

On the first day of the last week of the data-collection effort, tests were administered 
to pupils in all sample classrooms. Invigilators (test administrators) were trained and the 
evaluators circulated among the schools during the administration and checked on the 
appropriateness of the procedures. Several irregularities were found due to confusion among 
the invigilators. In three instances, the invigilators were trying to manage more than one 
classroom at a time, thereby leaving one class unattended. Although corrections were made 
when the supervisors arrived, it is possible that this allowed for the possibility of cheating. In 
fact, during the marking process, we were able to identify several indications of cheating and 
did not record the marks for those pupils. 

The marking sessions were held over the last four days of the study, with the same 
individuals used for invigilating also used for marking. However, two teachers and two SIP 
staff who had been trained by the KNEC to mark the Kiswahili exams were used. Although 
we were reluctant to use SIP staff for marking, the other invigilators hired were not qualified 
in Kiswahili. These were Form 4 leavers and university students. 
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Training was provided for the marking and spot checks was done for each set of tests 
for each school.. The marking of the open-ended items was done over the last two days and, as 
time ran out, less checking was possible. It is likely that there is less reliability in the scores 
on the open-ended items, although the results indicated such small variability within a school, 
and such great variability across some of the schools, that the significance of these differences 
would be likely to hold. 

Individual, unstructured interviews also were held with the Director and Associate 
Director of the Municipal Education Office and with three of the TAC tutors, two inspectors 
and one meeting of the SIP Advisory Committee was attended. At several points during the 
study, two or three of the senior researchers met with members of the SIP professional staff. 
These were unstructured sessions, but discussions centred around how they were trained, their 
ratings of their training, what they thought were the strengths and weaknesses of the 
programme, and what they would do differently if they had it to do over, or if AKF were to 
initiate a similar effort in another region. 

To attempt to gauge the quality and style of the training and classroom-based TA 
provided by the SIP Program Officers (POs), videos of training sessions were viewed. 
However, few of the videos actually showed the current staff of POs doing training. In several 
videos, the Project Director was doing the training and in others it was being done by POs 
who were no longer with the project or by outside consultants. One evaluator observed an 
hour of a live three-hour workshop on reading, and another observed several sessions where 
the POs met with teachers in their classrooms to provide coaching and TA . 

Description of Treatment and Control Districts 

Eleven treatment schools were drawn from SIP schools located within the Kisumu 
Municipality. However, it was not possible to obtain control schools from Kisumu since there 
were no schools that had not in some way been impacted by the SIP, especially with the high 
rate of teacher and head teacher transfers and the fact that SIP workshops were open to all who 
chose to attend, beyond those for whom the workshops were specifically organized. Although 
a number of schools had recently been added to the Kisumu boundaries, all were rural schools 
and so were determined to be less similar to SIP’s urban and peri-urban schools than those in a 
nearby district. Schools selected for control comparisons were drawn from a neighbouring 
community in Siaya, Kenya. Each of these communities is described below. 
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Kisumu Municipality: The Kisumu Municipality is located on the edge of Lake 
Victoria on Kenya’s border with Uganda. It is the largest town in Kenya and scheduled to 
become a city. It is the headquarters of the Nyanza Province, the commercial centre of 
Western Kenya. In 1994, the population of Kisumu Municipality was about 600,000, the 
majority belonging to the Luo ethnic group, with about 3,000 Asians. Several industries are 
located in Kisumu including: baking, fishing, brewing, pharmaceutical, and agricultural 
processing. A large number of the population are self-employed through fishing, agriculture, 
transportation, trading, bicycle and motor vehicle repair, and tailoring. 

Education in Kisumu: Pre-primary education is offered to children from 3 to 6 years 
of age and the number of pupils attending pre-primary school has grown from 7,406 in 1990 
to 9,638 in 1996. Primary education begins at the age of six and continues for eight years. 
Table 1 shows the growth of primary education in Kisumu by gender between 1990 and 1996 
which has grown by about 6,000 until 1994 when the Municipal boundaries were expanded. 
For the most part, Kisumu ’s enrolment growth since 1995 is due to expanded boundaries 
which brought in almost 100 small, rural schools. Table 2 shows the primary school 
participation rate by gender in 1994 in comparison with national rates. Kisumu s rate is 10 
percent higher than the national rate and just over one percent higher than Siaya’s. Table 3 
shows primary repetition and dropout rates. Kisumu has the lowest repetition rate at 10.2% 
compared with about 16% for Siaya and the nation. 



Table 1 

Primary School Enrolment in Kisumu Municipality - 1990-1996 



Year 



No. of schools 



Enrolment 



1990 

1991 

1992 

1993 

1994 

1995 

1996 



Public - Pvt. - total 



42 2 44 

42 7 49 

42 7 49 

42 10 52 



136 

123 



20 156 

13 136 



Boys 

14819 

15541 

14101 

15432 

23167 

28148 

29996 



Girls 

14904 

14789 

13811 

15069 

12738 

25482 

26346 



Total 

29723 

30330 

27814 

30501 

35905 

53930 

56342 



Source: Municipal Education Office - Kisumu 
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Table 2 

Primary School Participation Rates - 



Kisumu, Siaya, 


and National 


by Gender - 


1994. 




Boys 


Girls 


Total 


Gross Enrolment Rate (Kisumu) 


95.2% 


88.9% 


92.1% 


Siaya District (1994) 


91.9% 


89.4% 


90.7% 


National Gross Enrolment Rate 


83.2% 


81.6% 


82.4% 



Source: Ministry of Education, Statistics Section. 



Table 3 

Primary Level Repetition and Dropout Rates - 1994-1995 

District/Municipality Repetition Rate Dropout Rate 

Kisumu 10.2 5.8 

Siaya 16.8 4.5 

National 15.4 5.4 

Source: Ministry of Education, Statistics Section. 

The primary school curriculum: The curriculum is developed centrally by the Kenya 
Institute of Education and is the same for all treatment and control schools. It has a strong 
emphasis on practical skills and includes 13 subjects: mathematics, English, Kiswahili, 
Mother Tongue, geography/history/civics (GHC), music, science, art and craft, business 
education, agriculture, home science, religious education, and physical education. 

Siaya District: Siaya district is a one-hour drive north west of Kisumu with a 
population of about 2,000. It is primarily a peri-urban and rural district with 18 schools and 
an average pupil-teacher ratio of 1 to 19. The area is very dry with little rain and so does not 
support agricultural farming. Businesses include trading, bicycle and motor vehicle repair, 
tailoring, pottery, and blacksmith. The population is largely from the Luo ethnic group. In 
the Siaya sample, three of the schools selected were urban or peri-urban schools and one a 
rural school. Two were selected to compare with the top schools in Kisumu on the KCPE 
among the top hundred in the country - and the other two were taken to be average and could 
therefore compare with the average schools in Kisumu. 
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Selection and Sample 

Schools from Kisumu were selected for inclusion in the study by the evaluation team 
with the assistance of the SIP Project Director and the two Deputy Directors. The treatment 
schools selected for study were chosen based on several factors, but primarily to represent 
differing levels of time in the SIP project. Other factors included whether the teachers and 
head teachers who were originally trained at a school were still working at that school; head 
teacher longevity at the school; and schools where the treatment had been offered to both 
lower and upper primary teachers. In addition, the socioeconomic and geographic status of 
the schools was considered. Each factor is described in more detail below. 

Duration in SIP: The first selection cut was based on length of time a school could 
have been exposed to SEP influence. Since SIP had been operating for six years, those schools 
that had participated in the programme during the early years have had more opportunity to 
engage in SEP-sponsored workshops and to integrate the intended teaching. 

77: Group T1 represents schools that had the potential for the greatest amount of 
influence by SIP, having initially received training and coaching between 1990 and 1992 
during Phase 1, with possible ongoing participation in subsequent SIP workshops offered 
throughout Phase 2. This category also provides the opportunity to assess the durability of the 
SIP treatment. 

T2: Schools in group T2, began their SEP involvement in the early stages of Phase 2 
(September 1993), thereby having a three-year period of time to attend additional workshops 
and for the training and coaching to be internalized. This is referred to as Phase 2-3 years. 

T3: Group T3 schools had only participated in SIP during January through May 1996. 
Schools were not in session in April and the data collection began June, so their involvement 
as SEP schools has been 4-5 months, although some of these teachers or head teachers may 
have attended SIP-sponsored workshops over the years. The T3 group is referred to as Phase 
2-4 months. 

SEP had offered a series of workshops aimed primarily at head teachers and it was 
hoped that another category of “ head teachers only” could be identified and studied to assess 
the impact on teacher behaviour and school effectiveness when a school leader leams new, 
more effective strategies for managing a school. There is ample evidence supporting the 
influential role of the head teacher in school improvement, but we were not able to identify 
any schools where a head teacher had attended training, but teachers in that school had not, 
and so abandoned that category for study. 
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To summarize, the sample categories are: 



T1 / / 


Phase 1 (4 schools) 


T2 


Phase 2-3 years (4 schools) 


T3 


Phase 2-4 months (3 schools) 


C 


Control - Siaya (4 schools) 



Location: Most of the schools that were involved with SIP, were urban or peri-urban. 
To maintain consistency across the three treatment groups, only urban or peri-urban schools 
were selected for study. This is one of the reasons that Siaya was selected to serve as a control 
site, since it is a peri-urban area and all four of the control schools were peri-urban schools 
and, therefore, more similar to the treatment schools than would be a rural school in Kisumu. 

Based on these selection criteria, four schools were selected for each of groups T1 and 
T2, and three for T3. One school in each of groups T1 and T2 was a high-cost school and 
analyses reported later were conducted both with and without these schools to assess the 
impact of a more well-endowed school facility and parent population. SIP-trained teachers at 
the upper primary level in two of the T1 schools, all of the T2 schools, and two of the T3 
schools. All schools had head teachers that had been working at that school for at least one 
year, but most for several years. 

Instruments 

The following data was collected from all 17 schools described above - 11 treatment 
and four Siaya control schools and the two schools that had not participated in SIP but had 
benefited by attending many SIP workshops. All instruments were developed by the three 
senior members of the evaluation team and were pilot tested and revised during the last week 
of May 1996. Data collection began June 3rd and continued through June 20th. (Copies of all 
instruments used are in Appendix C.) 

Classroom observations: This instrument focused primarily on teacher and pupil 
behaviours associated with classroom management and child-centred teaching and learning. 
Although there is considerable variability in definitions and views of child-centred learning, 
the instrument was developed to document teacher use of the behaviours conveyed in the SIP 
training. Therefore, the SIP training manual - which consisted primarily of a collection of 
the handouts provided at the various workshops -- was the major source used to determine the 
behaviours to be observed. In addition, the observers documented the language of instruction, 
the language of pupil’s responses, the types of questions teachers asked and their responses to 
students’ answers. 
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In addition, the number of pupils with textbooks was counted (for English, maths, and 
Kiswahili), the number of exercises in their English and maths exercise books, use of learning 
aids/manipulatives, time off-task, teacher record keeping and assessment, and the presence of 
classroom learning materials. 

The observers took notes on the lessons and at the end of the observation, provided an 
overall rating and description of the lessons, some of which are described throughout this 
report. 



Household survey: A survey of households that had been used in several other recent 
studies was modified for use in the Kisumu context. This instrument was intended to capture 
estimates of family income, family education levels, familiarity with and attitudes towards the 
SIP programme; and whether the parents may have sent their child to the SIP school because 
it was considered a better school. 

Pupil interviews/questionnaire: The pupil questionnaire consisted of 33 items 

asking about the nursery school the child attended, language(s) spoken in various settings, 
school affect, locus of control, homework, reading, and TV watching. Fifteen pupils from 
each of the standard 3 sample classrooms were interviewed in their local language, while all of 
the pupils in the standard 6 observed classrooms were given the questionnaire to complete in a 
group setting after the observations had been completed. One of the observers was available 
to answer questions in the local language. 

School staff focus group: A focus-group interview protocol was developed for use 
with school staff. The interest was in trying to gauge the impact that SIP has had on the 
school as a whole and on the staffs’ views of SIP. There was also an interest in the role that 
the head teacher played as an instructional leader, so staff were asked about frequency of staff 
meetings, topics discussed at those meetings, frequency of head teacher s visits to the 
classrooms, and changes in the frequency and topics of formal and informal staff discussions 
about issues of teaching and learning. 

Head teacher questionnaire and teacher and head teacher interview protocols: 
In both teacher and head teacher instruments, the primary interest was of individual’s views 
and perceived impact of SIP. Instruments also included questions about parent involvement, 
views of the curriculum, language of instruction, teacher/head teacher academic and 
professional qualifications, perceived locus of control, and perceptions and definitions of 
child-centred teaching and learning. 

School climate checklist: Indicators of an effective school not captured through the 
instruments described above were included on a checklist to capture items such as, locks on 
classroom doors, windows that close, running water, cleanliness and attractiveness of school 
grounds and classrooms, number of teachers absent, etc. 
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Unstructured interviews: Unstructured interviews were held with the SIP staff, with 
the Municipal Education Officer, the Deputy MEO, and several TAC tutors. 

Tests: Tests were administered to standard 3 pupils in English and mathematics and 
to standard 6 pupils in English, mathematics, and Kiswahili. Each test is described below and 
a copy of the open-ended tests are in Appendix A. 

Standard 3 . The standard 3 English and mathematics tests consisted of two parts: 1) 
short-answer and matching items, and 2) one or two open-ended passages or questions . The 
format of the short-answer items was based on standard 3 English language and mathematics 
Kenyan national curriculum. The English test measured picture recognition, plurals, word 
endings, antonyms, past tense, use of “ a” or “ an” , and selecting the correct word to complete 
a sentence. The math test measured basic addition, subtraction, multiplication, division, 
fractions, and telling time to the five-minute interval. 

The open-ended passages had two purposes - to show how tests can be designed to 
support child-centred teaching and to measure the more central, integrated skills within each 
of the two disciplines. The English-language open-ended passage was intended primarily to 
test pupils’ ability to read and write in English. It used simple words to account for the fact 
that English was not their Mother Tongue, and was intended to show how practice tests could 
promote the teaching of writing, encourage creativity and imagination, and validate children’s 
views and feelings. It was also intended to convey how a writing exercise could be fun for 
children - all important aspects of child-centered teaching. Below is the open-ended English 
passage: 

Read die story and_ answer the question. Write as much as you can. 

Birds can fly. If you could fly, where would you go? What would you do when you 
got there? Draw a picture of you flying. 

The two open-ended mathematics questions measured whether pupils can 
communicate and represent mathematical concepts, whether they understand basic 
mathematical operations when represented in a word problem, whether they understand 
Kenyan currency, and were written to “connect” to their lives. The open-ended questions 
were: 



Draw a picture to show your fnend how many 10 cent coins are in a shilling. 

Draw a picture that shows that you have three more sweets than your brother. Your 
brother has 6 sweets. 
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Standard 6 - English. The standard 6 English and math tests used in this study had 
been developed, and used in the evaluation of a primary school improvement intervention in 
Busia — a town about two hours drive from Kisumu. Both were based on the National 
curriculum and each consisted of two parts: 1) a multiple-choice test of 50 items, 2) an 
open-ended question or passage. 

The open-ended mathematics question was added by the SIP evaluators and had three 
parts, although we later discovered some problems with the question. 

Pretend that you are going to plant a garden of maize. The garden is 3 meters long by 
3 meters wide. You have 15 maize seeds and each maize plant will need 1 meter of space to 
grow. 

Draw a picture to show the garden plot and how you would arrange the maize seeds 
so that each one has 1 meter of space. 

Will you be able to fit all 15 maize seeds into the garden plot? 

If not, how many will be left out? 

Standard 6 - Kiswahili. No multiple-choice or short-answer questions were given in 
Kiswahili -- only an open-ended question designed to measure if the pupils could read a 
simple passage written in Kiswahili and write a response in Kiswahili. The English and 
Kiswahili passages are in Appendix A. 

The Kiswahili markers were trained markers for the Kiswahili portion of the KCPE. 
Initially, they proposed to use the marking scheme used for marking English and Kiswahili 
compositions in the KCPE which consisted of the following scales and point allocations: 
reading - 2 points; comprehension - 2 points; writing, spelling, punctuation relevancy, 
vocabulary, and grammar - 6 points; total points possible - 10 

The marking scheme that we ultimately used is quite different and much more 
detailed. It was deliberately used to illustrate the possible instructional value that it would 
have in communicating the types of strengths and weakness pupils have in writing and in 
making suggestions for teachers to improve the teaching of writing. (Actually, the Kiswahili 
markers, who often mark the KCPE Kiswahili examinations, found this detailed marking 
scheme pedagogically useful and indicated that they planned to suggest it be used in KCPE 
marking sessions.) The marking schemes are reported in Appendix B. Table 4 is a summary 
of the instruments used in this study, the target groups interviewed, observed or tested, and the 
sample size for each group. 
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Table 4 

Summary of Instruments, Target Groups and Sample Size 

Instrument Target Group Sample Size 

Classroom observation Standard 3 and 6 teachers 34* 

Teacher Interviews Standard 3 and 6 teachers 34 

Head Teacher Interviews Head teachers 17 

Head Teacher Questionnaires Head teachers 17 

Parent Interviews Parents of pupils tested 510 

Pupil Interviews (Std. 3) Pupils in standard 3 sample 255 

Pupil Questionnaire (Std. 6) Pupils in standard 6 sample 634 

School Committee Interviews School Committee Representatives 17 

Focus Groups School Staff 17 

English and Maths Tests Pupils in standard 3 sample, 

(Std. 3) 617 

English, Kiswahili and Pupils in standard 6 sample, 

Maths Tests (Std. 6) 610 

*The two non-SIP, but high workshop-attendance schools were not included in analysis. 
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CHAPTER 3 
FINDINGS 



The ultimate beneficiaries of SIP’s efforts to improve the skills and behaviour of 
teachers are the children who pass through these primary schools. In the Phase 1 evaluation, 
the authors concluded that student learning had “ not declined as a result of the intervention” 
and reported some evidence of gains in language skills. This section reports on whether the 
second phase of SIP’s work with teachers resulted in increases in student learning beyond 
those found in the control schools. Part of this analysis attempts to correlate teachers’ child- 
centred teaching and classroom-management behaviours, as measured during the classroom 
observations, with student learning. This is followed by a detailed description of the teaching 
behaviours observed to assess whether SIP really did contribute to more child-centred 
teaching in the project schools, as compared with the control schools. A later discussion 
evaluates the quality of the implementation of SEP and the extent to which the project’s 
objectives, as articulated in the project proposal, were met. 

To What Extent Did SIP Impact on Student Learning? 

The analysis of test results was conducted in two stages: an analysis of variance 
(ANOVAs) to detect simple differences in performances across groups, and regression 
analyses to control for external factors that may contribute to achievement results but are not 
attributable to the SIP treatment, such as family income and parents’ education. 

At the time the tests were given in mid-June 1996, students had received about five 
months of instruction, so one would expect that students had not mastered all of the content 
for that year. Two of the treatment groups (T1 and T2) contained a high-cost school, but none 
were in the T3 or control schools. It is likely that the parents of pupils at the high-cost schools 
have higher incomes and educational backgrounds, may be more likely to read to their 
children, help them with homework, have books and magazines in the home, and provide 
textbooks and other instructional materials for their children. It is also more likely that classes 
in these schools are smaller and have other instructional inputs that may enhance learning. All 
of these factors related to being in a high-cost school may serve to increase test scores beyond 
those of low-cost schools and beyond what SIP may have contributed, so the reporting of test 
results shows comparisons between the treatment and control schools with and witho jjt the 
two high-cost schools. In addition, the regression analysis, described later, holds these and 
other factors constant to assess their influence on test scores in relation to the influence of the 
SIP treatment. 
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Overview of test results without controls for external factors 



Repeatedly, the test results show that pupils in T1 schools performed significantly 
better than all other groups, followed closely by pupils in the T2 and/or control schools. The 
T3 schools — those that have been in SIP for only 4-5 months -- performed worse than the 
other treatment schools and almost always less well than the Siaya control schools (See Table 
5). 



Table 5 

Summary of Test Scores by Class, Subject and Question Type 
with and without High-cost schools (HCS) 

Mean Percent Correct and (Standard Deviations)* 



Test 


T1 


T2 


T3 


Control 


Total Mean 


Standard 3 (N) __ . . 


Eng. SA (617) 
w/o HCS (588) 


79 (17)*** 
76 (16)*** 


64 (24)** 
53 (21)* 


45 (29) 


51 (26) 


60 (27) 
52 (29) 


Eng. OE (601) 
w/o HCS(588) 


28 (24)*** 
19 (16)*** 


9(15) 

4(8) 


4(8) 


12(13)* 


13 (18) 
10 (13) 


Math SA (610) 
w/o HCS (604) 


59 (19)*** 
57 (18)** 


51 (18)* 
45 (16)* 


44 (17) 


52(15)* 


52 (18) 
48 (17) 


Math OE (615) 
w/o HCS (607) 


18 (26)*** 
12 (15) 


9 (16)* 
5(11) 


3(8) 


9(13)* 


10 (18) 
7(12) 


Standard 6 (N) 


English MC (610) 
w/o HCS (630) 


60 (22)** 
55 (20)** 


54 (23)* 
43 (17)* 


43 (18) 


51 (19)* 


52 (21) 
46 (19) 


English OE (274) 
w/o HCS (238) 


18 (24)** 
14 (22)** 


14 (20)** 
13 (23)** 


4(10) 


3(7) 


11 (19) 
9(18) 


Math MC (543) 
w/o HCS (546) 


43 (15)** 
39 (14)* 


40 (17)* 
32 (11) 


33 (12) 


37(11)* 


38 (14) 
35 (12) 


Math OE (546) 
w/o HCS (546) 


9 (10)* 
7(7) 


9 (10)* 
6(6) 


5(6) 


7(8) 


8(9) 

6(7) 


Kiswahili OE (287) 
w/o HCS (299) 


36 (25)** 
45 (21)* 


28 (17)* 
25 (17) 


26 (22) 


18 (15) 


27 (21) 
26 (21) 



*** Means that this group performed significantly better than all other groups. 

** Means that this group performed significantly better two other groups. 

* Means that this group performed significantly better than one other group. 

All reported tests of significance are at the p < .10 level unless otherwise noted. 

However, one of the T3 schools achieved a mean score of only 8 points on the English 
short-answer test, bringing the average score down substantially for group T3. The other two 
schools in T3 acquired 41 and 71 points on this test. If this school’s score were removed from 
the analysis for the English short-answer test, the mean for T3 would be 55, higher than the 
mean for the control group, but still lower than the mean for T1 and T2 on this test. 
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One explanation for the superior performance of the T1 (Phase I) schools is that the 
teachers continued attending SIP workshops during Phase 2 and so received more training and 
reinforcement than did Phase 2 teachers. In fact, SIP records support this hypothesis. 
Teachers in three of the four T1 schools attended a total of 1 15 workshops during Phase 2. 

Standard 3: At the standard 3 level, the Siaya control schools, which did not receive 
any special project staff development during the six-year period of the SEP treatment, 
performed better than the T3 pupils on all tests and the same as T2 pupils on one test and 
better on two tests. The T2 pupils only outperformed the control schools on the English short- 
answer test, and their edge disappeared when the high-cost school was removed from the 
analysis. Although T1 schools performed significantly better than all other groups on all tests, 
when the high-cost school was removed, there are only two tests where they significantly 
outperformed control schools - the English short-answer and open-ended tests. In short, then, 
there appears to be little achievement advantage to being in a SIP school until a school has 
been in the project for at least five or six years, and only on the English tests. And, this 
advantage may be due in part to higher numbers of workshops attended. 

Standard 6: Scores were lower overall at the standard 6 level, although the tests for 
this level were more difficult than those for standard 3. Across all tests except math open- 
ended, T1 schools performed significantly better than T3 and control schools, but never 
significantly better than T2 schools. T2 schools only performed significantly better than 
control schools on the English open-ended test, although their scores on this test were only 14 
percent. The T3 schools consistently performed significantly worse than the T1 and T2 
schools and usually worse than the control schools. 

Comparing levels: In both subjects, the standard 6 pupils performed less well than 
did the standard 3 pupils. Scores on the English short-answer/ multiple-choice tests averaged 
60 percent across all groups at standard 3 compared with only 52 percent at standard 6. On 
the math short-answer/multiple-choice tests, average scores were 52 percent for standard 3 and 
only 38 percent for standard 6 pupils. Scores declined slightly and variously at both levels 
when the high cost school was removed from T1 and T2 groups. Scores across all groups on 
the open-ended tests were very low and are explained more fully below. 

Performance on open-ended questions: Pupils in all groups performed poorly on all 
open-ended tests -11 percent or lower across groups. On tests that required pupils to read 
and write in English, most either wrote nothing at all or simply rewrote the prompt. 
Performance was somewhat better in Kiswahili, but still poor (average of 27%). 
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English 3: Pupils in all groups performed very poorly on the open-ended question in 
English. The total possible points was 14, with 10 points allotted to a written response to the 
prompt and four points to drawing a picture. Treatment group 1 obtained a mean of 3.9 points 
out of 14 (27.6%), which was significantly better than any of the other groups but still poor. 
The Siaya control group performed better than both the T2 and T3 groups. Since almost 
one/third of the 14 points were allocated to drawing a picture, it is likely that the scores for 
reading and writing are even lower than the scores recorded in Table 5. 

What is particularly notable in these results is that many pupils received quite high 
scores on the short-answer questions but could not demonstrate that they could read and write 
in response to a very short, simple prompt. During the classroom observations, we observed 
that when teachers were teaching English, it was usually a grammar lesson. Only a couple of 
the teachers actually had the children read, and that simply involved children taking turns 
standing up and reading a paragraph. They weren’t asked to predict what the story or passage 
might be about, to connect it to their lives or what they already know, to summarize a story, or 
to create their own stories. We saw no standard 3 teacher teaching children to write, other 
than to do the exercises in their textbooks. The exercises are generally fill-in-the-blank or 
matching and do not involve the pupils in communicating; in generating sentences, ideas, or 
thoughts; or in expressing their own views and feelings. Below is a sample of the range of 
responses to the English reading and writing prompt. 

Prompt (Standard 3) 

Read the story, and answer the question. Write as much as you cotl 

Birds can fly. If you could fly, where would you go? What would you do when you got there? 
Draw a picture of you flying. 

Pupil 1 (Score = 1 for written passage; 3 for the picture) 

I cant fly. 
ane kiat canfly 
I cana fly above 
aboat can flia 
I can fay 
if yianf lay 
I cane slay 

This pupil answered 29 of the 36 short answer questions correctly and was able to 
draw a picture of himself flying. 
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Pupil 2 (Score = 8: 4 for picture). 

I’d go to england 
I’d go to calofonia 
and also america 
then to atlanta* 

I’d go to atlanta 

for skhing playing with 

ice and having a lot more fun 

♦This test was given just before the Olympic games in Atlanta. 

This pupil answered 35 of the 36 short-answer questions correctly and was from a 
high-cost school. 

Pupil 3 (Score =1: 0 for picture) 

This pupil must have recognized the words “ birds” and fly.” 

birds 

hen 

I sing 

Iypiay 

batafigns 

duke 

ositrh 

She answered 23 of the 36 short-answer questions correctly but also drew a bird rather 
than a picture of herself flying. 

Pupil 4 (Score = 9: 4 for picture). 

If I could fly, I could go 

high up in the sky and see the 

earth below me. I could see the 

heaven and I could worship 

the lord my God. I could get 

blessings from God and see 

how he could handle the 

worle world by his power, 

and how he destroyed the evil singl 

handedly. And there is no more that the end. 
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This pupil answered all of the short-answer questions correctly and drew a colorful 
picture of himself flying in a superman costume and wrote, “I am superman!, flying like a 
rocket!” He also was from a high-cost school. 

Puoil 5 (Score = 6: 4 for picture). 



I can’t fly 
if I could 
fly I could go 
where I want to go. 

Birds can fly. 

This pupil answered 35 of the 36 short-answer questions correctly and drew a 
colourful, charming picture of himself flying with the words “ this is me I am flying” next to 
his drawing — also from a high-cost school. 



Pupil 6 (Score = 2: 1 for picture) 



This T1 pupil only answered 20 of the 36 questions correctly but some of her passage 
was coherent. (It was difficult to tell if there were real spaces between the words.) 



I am fyimg to goin the god 
and bird is fiyiny to go in the god 
I am going in your mother in home 
I am fly to gothomonog 
I an gathr it griant gthora 
Ianiohatgtritognhat 
Iaragiriohnoagtfiahi 
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Mathematics open-ended: The S3 pupils also performed very poorly on the two open- 
ended questions for mathematics. Out of a possible 10 points, most groups received less than 
one point. Pupils in standard 6 received less than one point out of 12 possible points for their 
responses to the open-ended questions. When compared with their short-answer and multiple- 
choice test scores, these results suggest that pupils can perform some basic mathematical 
calculations but do not know how to apply these skills to solve real-life types of problems, nor 
represent basic mathematical ideas. The open-ended questions and two successful responses 
are below. Most simply provided no response: 

Prompts (Standard 3) 



Draw a picture to show your fnend how many 10 cent coins are in a shilling. 
Draw a picture that shows that you have three more sweets than your brother. 



Your brother has 6 sweets. 




'you have^fliiree onorc sweets than 
your brother. Your brother Has# 
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English and- Kiswahili essay questions : Scores on the standard 6 essay questions averaged 11 
percent for the English passage and 27 percent for the Kiswahili passage, although pupils in T1 
did score significantly higher than any other group. In these tests also, many students wrote 
nothing or rewrote the prompt. (See Appendix A for the passages and Appendix B for the 
marking schemes.) It is both interesting and puzzling to compare scores for English and 
Kiswahili by school. Table 6 shows the total scores for English and Kiswahili compositions by 
school and treatment group, as well as the difference between the scores for each school. Several 
interesting findings are clear from this table. First, the scores on the Kiswahili compositions are 
higher overall than the scores on the English composition. However, there appears to be no 
correlation between a pupil’s ability to read and write in English or Kiswahili. The difference in 
a school’s marks between the two tests ranges from two to 47 points, and in only two instances is 
the English score higher than the Kiswahili score (differences are listed as a negative value when 
the English score is higher). 



Table 6 

Comparison of Mean Scores (percent) for English and Kiswahili Composition by School 

and Treatment - Standard 6 



School English Composition Kiswahili Composition Difference in Scores 



Treatment 1 


1. A (HC) 


28 (28) 


8(9) 


-20 


2. B 


8(9) 


55 (19) 


47 


3. C 


10 (18) 


37 (20) 


27 


4. D 


24 (31) 


42 (19) 


18 


Treatment 2 


1. E 


11 (15) 


36 (14) 


25 


2. F (HC) 


20 (6) 


37 (14) 


17 


3. G 


26 (32) 


17 (14) 


- 9 


4. H 


3 (5) 


21 (17) 


18 


Treatment 3 


i.i 


5(5) 


7(12) 


2 


2. J 


5(13) 


43 (18) 


38 


3. K 


4(12) 


28 (22) 


24 



Control - Siaya 
l.L 


5(5) 


2. M 


3 (4) 


3. N 


4(9) 



23 (15) 18 

16 (17) 13 

8(9) 4 
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Second, there is tremendous variability within each of the treatment groups, with 
English scores. ranging 20 points and Kiswahili scores ranging 47 points within group Tl. 
Interestingly, the school that received the highest scores on the English test received one of 
the lowest scores on the Kiswahili test. This is likely due to the fact that this school has a high 
population of Asian pupils. However, the other high-cost school that also has a high 
population of Asian pupils performed substantially better on the Kiswahili composition. Two 
of the treatment schools received very low scores on the English test yet their Kiswahili scores 
were both 47 points higher. It is possible that the differences in scores was in part due to use 
of different markers, but since they were using a common marking scheme, significant 
differences are unlikely. The test passages were very similar in style, format, vocabulary and 
difficulty level, so are not likely to be the cause of these differences. 

Summary of Test Results 

In summary, students in standard 3 performed better than students in standard 6 on all 
tests, and scores on the English tests were higher than scores on mathematics tests. Students 
performed reasonably well on the short-answer and multiple-choice tests, suggesting that they 
are learning what is in the curriculum in the way that it is taught in the curriculum and the 
textbooks. However, there is clear evidence that they are not learning how to read, write, or 
communicate in English or in Kiswahili. Nor are they learning to apply simple mathematical 
concepts and skills to real-life types of problems. 

Overview of test results with controls for external factors 

The test results reported above are limited in that they do not take into account a 
variety of factors that can influence student learning, such as parents’ income and education, 
the teacher’s experience, the pupil-teacher ratio, textbooks, etc. The next section uses 
regression analysis to control for these factors in determining if SIP had an impact on test 
scores. Two of the variables used in the regressions are indices of child-centred teaching and 
classroom management. The explanation for how these indices were arrived at is explained in 
the section following the regressions. 
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In each of these regressions, the dependent variable is one of the sets of test scores and 
the independent variables either tried and/or used include: 1) family characteristics - family 
income, proportion of income spent on food, land owned in the village, mother’s level of 
education, whether the parents read to their child, whether they help the child with homework, 
and hours of TV watching); 2) teacher characteristics - academic qualifications, number of 
years of teaching experience; 3) pupil characteristics - age, gender, nursery school attendance, 
quality of nursery school; 4) classroom characteristics - class size, percent of pupils in the 
class who own a textbook, whether the parents pay the teacher to tutor their child; 5) treatment 
characteristics - treatment group, number of workshops attended by teachers and head 
teachers, extent of use of child-centred methods and classroom management behaviours. 
Detailed explanations of the regressions are in Appendix C and a summary of the findings is 
below. 



Family characteristics: Percent of income spent on food and the amount of land 
owned in the village were both tried as proxies for family income or wealth. However, the 
variable that proved most robust was an index of parents’ source of income. During 
interviews with parents, they were asked to indicate how they earned their living. Their 
responses were coded to represent income levels — i.e., l=very poor, 2=poor, 3=working 
class, 4=middle class, 5=wealthy. This code was then entered into the regression equation. 
The income distribution for Kisumu is almost normal - surprising in a developing-country 
context. However, the sample schools were all either urban or peri-urban and it is likely that a 
larger percentage of poorer families would be found in the rural areas. 

Both mother’s and father’s levels of education were tried in the analyses and found to 
have a positive influence in some regressions but not in others. Ultimately, mother’s education 
was used. Three other variables were tested and found not to be significant predictors of 
scores on these tests and so were not used in the regressions reported: whether someone in the 
household helps the child with homework; whether a parent reads to the child (although not 
significant here this variable did have a positive influence on achievement); and the number of 
hours the child watches TV - either at home or at a neighbour’s. Interestingly, many of the 
children indicated that they watched TV, and the SIP staff reported that some may watch at a 
neighbour’s house if their family didn’t own a TV. 

Teacher characteristics: Teachers’ and head teachers’ academic qualifications in 
terms of the number of years they had attended school were tested and found not to be 
significant and so were dropped. This is likely due to the small amount of variance in the 
amount of education among teachers. Since all teachers are required to attend two years of 
pre-service teacher training, this was not used as a variable. The number of years a teacher 
had been teaching was a significant factor in pupils’ test scores in most regressions and was 

used throughout. 
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Pupil characteristics: Four pupil variables were tried: age, gender, nursery school 
attendance, and- the quality of nursery school attended. Nursery school quality was tried 
because SIP staff indicated that the quality varied considerably and was perceived by the 
community as being an important factor in pupils’ success at the early grade levels. Pupils 
were asked to indicate the nursery school they attended and the nursery schools were then 
rated on a 1-5 scale for quality by the MEO staff person responsible for overseeing nursery 
schools. However, there were not sufficient responses from pupils so the variable was 
dropped. Almost all of the standard 3 pupils had attended nursery school so there was little 
variability in attendance at these grades and it too was dropped. The significance of the 
variables age and gender varied across tests - significant in some and not in others. 

Classroom characteristics: Pupils in Kenya are required to provide their own 

textbooks. Textbook was calculated as the percent of pupils in the class who had a book. 
However, we discovered that textbook is highly correlated with mothers’ education and family 
income so often becomes insignificant when these variables are in the equation together. In 
fact, the simple correlations of textbook with test score are fairly low - .18 to .29. Parents’ 
income and education appear to be determinants of textbook ownership since pupils must 
purchase their own books. Therefore, textbooks are only occasionally included in the 
regressions, despite their obvious influence on learning. Class size was tried but found to be 
insignificant when other variables were in the equation. When only class size and treatment 
group were included, class size was significant, but in a direction that is counter to what one 
would expect - larger classes predicted higher scores. One classroom in the T3 group had 95 
pupils and high scores on all tests, so it was removed from the regression to see if it was 
distorting the results - class size was still not significant. The percent of pupils whose parents 
pay their teacher to tutor outside of class was used and found not to be an influence in test 
scores. 



Treatment characteristics: The number of professional development workshops 

teachers attended was often significant. These workshops included SIP and SPRED 
workshops and those sponsored by the TACs, although most of the workshops were SIP- 
sponsored. The two indexes created from teacher behaviour observed during the classroom 
visits were used as treatment variables - child-centred teaching and classroom management - 
since these were target behaviours of SIP training and TA. (See the later discussion in the 
next section for how these indexes were created.) However, it should be noted that teachers 
may use these behaviours without SIP influence, so they also could be considered non- 
treatment variables. 
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A note on sample size: Although the number of test scores available is high - around 
620 per class (grade), the number of cases in the following regressions is only about 15% of 
that sample. This is because the data included in the regressions are gathered from several 
sources — teacher interviews, pupil interviews, parent interviews and classroom observations. 
In order for a student’s case to be calculated in the regression equation, the relevant data must 
be available for that student from all of these sources. If even one piece of information is 
missing, that case is dropped from the analysis. Regressions were not done for the open- 
ended and composition tests since scores for all groups were less than 28 percent and 
generally less than 20 percent. 

Findings - Summary 

Standard 3: When family, teacher and school variables are notconsidered, the SIP T1 
and T2 pupils performed significantly better than T3 and control pupils in the English short- 
answer tests, but only T1 pupils performed significantly better in the math short-answer test. 
And although the T1 pupils performed significantly better in the open-ended tests, their scores 
were so low as to not be passable. 

However, when family income and mother s education are controlled for, being in a 
SIP T1 school is no longer a significant contributor to learning achievement, but being in a T2 
school does appear to have a positive influence on test scores in both English and math, 
although not significantly in English (see Table 7). Although T2 is not significant, the 
coefficient is 12.4. This means that a pupil’s score is likely to increase by 12.4 percentage 
points because he or she is in a SIP T2 school. However, the coefficient for the T1 schools 
drops from 21.6 to only 3.1, indicating that workshops attended and classroom management 
account for a much larger effect on T1 schools than these factors do in the T2 or T3 schools. 
A similar effect occurs in the mathematics test scores. The T3 coefficient increases 
considerably when the number of years a teacher has been teaching is added, suggesting that 
these schools have teachers with less experience than do T2 schools. This means that progress 
being made in a T3 school is more likely due to SIP than it is to an experienced teacher. 
Workshops attended was a significant and positive influence on pupil performance in both the 
English and mathematics tests. Test scores are estimated to increase by over two percentage 
points for each workshop a teacher attended and the T1 teachers attended significantly more 
workshops than did other teachers. Child-centred teaching was not a significant contributor to 
test scores. Classroom management behaviours were a significant influence on English scores 
but not on math. The most consistently influential variable on test scores is the number of 
years a teacher has been teaching. 
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Regression Table 7 
Short-answer Tests - Standard 3 



Depend.Var. 




English 






Mathematics 


Equation 


1 


2 


3 


4 


1 


2 


3 


Constant 


42.0 


4.0 


-4.4 


-45.5 


47.1 


29.2 


19.0 




(.00) 


(.63) 


(.82) 


(.00) 


(.00) 


(.00) 


(0.3) 


T1 


36.8 


22.9 


21.6 


3.1 


11.8 


9.9 


11.8 




(.00) 


(.00) 


(.04) 


(.57) 


(.05) 


(.04) 


(.07) 


T2 


22.3 


14.1 


19.6 


12.4 


3.5 


2.1 


11.9 




(.07) 


(.11) 


(.07) 


(.12) 


(.58) 


(.63) 


(.01) 


T3 


2.6 


1.51 


8.6 


4.6 


-3.2 


-8.2 


2.5 




(.87) 


(.85) 


(.54) 


(.35) 


(.65) 


(.01) 


(.65) 


Family 




7.43 


5.4 


3.1 




3.7 


1.2 


Income 




(.01) 


(.04) 


(.17) 




(.03) 


(.23) 


Mother's 




6.20 


3.5 


1.8 




2.6 


0.6 


Educ. 




(.00) 


(.08) 


(.24) 




(.00) 


(.58) 


Years 






1.4 


1.5 






1.6 


teaching 






(.02) 


(.00) 






(.00) 


Workshops 








2.13 








Attended 








(.00) 








Child-centred 








-.61 








Teaching 








(.01) 








Classroom 








3.3 








Management 








(.00) 








Adi R2 


0.26 


.50 


.40 


.41 


.07 


.29 


.27 


N 


688 


106 


83 


83 


703 


107 


85 



4 

4.2 
(.65) 
5.0 
(.37) 
10.4 
(. 01 ) 
2.8 
(.53) 
0.6 
(.53) 
- 0.2 
(.88) 

1.2 
(. 00 ) 
0.7 
(.05) 
- 0.1 
(.52) 
0.8 
(.33) 

.28 

82 



Note: Figures in parentheses are probabilities that the t-statistic is likely to be greater than the value estimated 
(but not shown here). Standard errors have been adjusted for heteroscedasticity and clustered sampling within 

schools. 



Standard 6: When family, teacher, and school variables are not_considered, the SIP 
T1 test scores were significantly greater than the T3 and control schools in all tests but math 
open-ended (see Table 8). There was no significant difference between T2 and control 
schools. However, when family variables are considered, SIP had no influence on student 
learning at the standard 6 level as measured by these tests. The number of years a teacher has 
been teaching was not significant as it was at the standard 3 level, and the other statistics for 
the other variables were unreliable for all standard 6 tests and all open-ended tests at both 

grades. 
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The child-centred teaching behaviours promoted by SIP did not seem to have a 
positive influence on test scores at either grade level. 

Regression Table 8 
Multiple-choice Tests - Standard 6 



Depend. Var. 




English 


Mathematics 


Equation 


1 


2 


3 


1 


2 


3 


Constant 


45.3 


16.9 


4.6 


35.9 


15.0 


16.7 




(.00) 


(.08) 


(.72) 


(.00) 


(.00) 


(.00) 


T1 


14.3 


2.7 


-2.7 


7.1 


5.5 


3.0 




(.06) 


(.69) 


(.68) 


(.05) 


(16) 


(.43) 


T2 


9.1 


2.1 


-3.2 


4.3 


1.6 


-1.6 




(.39) 


(.75) 


(.71) 


(.51) 


(.72) 


(.77) 


T3 


2.7 


0.1 


8.9 


-3.2 


1.7 


-1.9 




(.72) 


(.98) 


(.29) 


(.31) 


(46) 


(.72) 


Family 




5.2 


7.8 




3.8 


5.5 


Income 




(.11) 


(.00) 




(.03) 


(.00) 


Mother's 




5.4 


4.2 




3.1 


3.3 


Educ. 




(.00) 


(.01) 




(.01) 


(.02) 


Years 






.83 






-.4 


teaching 






(.17) 






(.50) 


Adi R2 


.09 


.25 


.24 


.06 


.22 


.21 


N 


711 


112 


96 


639 


99 


85 



Note: Figures in parentheses are probabilities that the t-statistics is likely to be greater than the value estimated 
(but not shown here). Standard errors have been adjusted for heteroscedasticity and clustered sampling within 

schools. 

Table 9 

Summary of Regressions - SIP Influence Significant (Yes/No) 



Test 



Mean % Mean % In SIP* Workshops 



CCT** CM*** 



Standard 3 
Eng. SA 
Math SA 

Standard 6 
Eng. SA 
Math SA 



wHCS 

60 

52 

wHCS 

60 

52 



w/o HCS 


52 


T21 


Yes 


No 


Yes 


48 


T2 


Yes 


No 


No 


w/o HCS 


52 


No 


No 


No 


No 


48 


No 


No 


No 


No 



Regressions for open-ended tests were unreliable and are not shown. 1 Significant at p <.12 
* Participation in SIP ** Child-centred teaching behaviours *** Classroom management behaviours 
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KCPE Results 



At the end of the eight-year primary cycle, pupils take the Kenya Certificate of 
Primary Education Examination (KCPE) - a test that is used to provide certification of 
successful completion of primary school and to select pupils for admission to secondary 
school. It is reasonable to ask whether SIP had any influence on performance on this 
examination, but, if there was an influence, it would have been quite limited since SIP did not 
begin working with teachers in standards 4-6 until late 1994 and the KCPE is administered to 
pupils in standard 8. Even for the three schools that began participating in SIP in 1990, only 
the standard 3 pupils would have reached standard 8 by 1995, and pupils in only two of these 
schools would have had SIP-trained teachers in standards 4-6 and none would have had SIP- 
trained teachers in standards 7 and 8. One of these two is the Aga Khan Primary School, a 
high-cost school which has traditionally scored at the top of all schools in Kisumu. 

Given these caveats, Table 10 shows several interesting findings: Kisumu ’s scores on 
the KCPE were 43 points higher than Siaya’s in 1989, but only 35 points higher in 1995. This 
suggests that Kisumu’s schools (or pupils) were better than Siaya’s before the SIP intervention 
and that the disparity between the districts decreased in Siaya’s favour. In addition, Kisumu’s 
ranking on the KCPE, compared to schools across Kenya declined from 7th in 1989 to 16th in 
1994, suggesting that if SIP did have an influence, it would have been negative. 

Table 10 

Mean KCPE Performance - Kisumu and Siaya 

Kisumu Siaya Difference 





Mean 


National 


Mean 


National Mean score 






score 


Rank 


Score 


Rank 




1985 


315.94 


10 








1986 


315.994 


12 








1987 


311.01 


12 








1988 


311.66 


12 








1989 


369.61 


7 


326.02 


33 


+43 


1990 


365.76 


11 


330.00 


34 


+36 


1991 


364.47 


11 


344.35 


24 


+20 


1993 


368.42 


13 


332.48 


32 


+36 


1994 


368.41 


16 


333.41 


33 


+35 



Source: Ministry of Education, KNEC. Total number of districts ranked is 59 in 1995; 46 in 1991/ 92. 
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What Difference Has SIP Made in Enrolment, Attendance and Class Size? 

Appropriate data were not available to assess SIP’s impact on these factors. The MEO 
was not able to provide school enrolment figures for years before 1993, and generalizations 
about SIP’s impact based on changes in enrolment also would be questionable due to a high 
rate of AIDS-related deaths in Kisumu. However, enrolment rates are quite high in Kisumu — 
92% - about 10% higher than the national rate and 1.5% higher than Siaya’s rate. 

Girls were enrolled in slightly higher numbers in groups T2, T3 and the Siaya control 
schools, with little difference between the genders in T1 schools. On the day of the 
observations, pupil absences averaged about five in Tl, T2, and the Siaya control schools, but 
were substantially higher in the T3 schools with an average of 17.25 pupils absent. Class size 
was also much higher in the T3 schools, with an average of 63 pupils per class compared with 
44 in the Tl schools, 41 in the T2 schools and 50 in the control schools. The larger class size 
may have contributed to absences. The T3 schools also had a higher proportion of students 
with more than two weeks of absences. 

Did SIP Have an Impact on Teaching and Learning? 

There were several study questions that addressed the impact of SIP on teaching and 
learning: 

■ How effective were SIP teachers in implementing child-centred teaching and learning 
in their classrooms? 

■ Is there a correlation between teachers’ facility at child-centred teaching and students’ 
academic achievement? (Answered in previous section.) 

■ What are the project teachers’ level of understanding regarding child-centred teaching 
and learning? 

■ To what extent did the project impact on teachers’ and head teachers effectiveness? 

Each question is addressed in order. 
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How effective- were teachers in implementing child-centered teaching and learning in 
their classrooms? 

Procedure. The classroom observation instrument was designed to assess the 
dimensions of child-centred teaching and classroom-management strategies contained in the 
SIP workshop materials. One standard 3 and one standard 6 classroom were observed by two 
evaluators in each of the treatment and control schools. Two lessons were observed for each 
class and in most cases, the lessons observed were English and mathematics. In the upper 
grades, several science lessons were observed. Generally, at the lower-primary level, standard 
3 classes were observed, but in two cases, the standard 3 teacher was not available and was 
replaced by a standard 2, SIP-trained teacher. However, in all cases, the standard 3 SIP 
classes were tested. Also, for some of the upper-primary observations, the SIP teacher (when 
there was one) was not teaching a core subject (i.e., English, math, science or social studies) at 
standard 6, in which case, the teacher was observed teaching at a lower class level in a core 
subject. Again, it was that teacher’s standard 6 core subject class that was tested. 

The definition of child-centred teaching as used in this evaluation was derived from 
various sources, including the project proposal, the handouts provided to teachers at 
workshops, and, through discussions with SIP staff. 

In the following discussion, teachers’ behaviours are reported in three ways: a brief 
overview of the typical teacher behaviour found in most schools; indexes of child-centred 
teaching and classroom-management behaviours comprised of a collection of related 
classroom behaviours; and more detailed descriptions of individual teaching behaviours that 
are considered to be aspects of child-centred teaching. 

An overview of the typical teacher behaviour observed: Overall, we did not 
consider the teaching in the SIP classrooms to be particularly child-centred. There was 
remarkable similarity across teachers. Most lessons began with a 10-minute explanation of 
the topic, with teachers asking a rapid series of closed, factual questions that did little to 
engage pupils in higher levels of thinking or reasoning. An assignment was then given, pupils 
collected their exercise books, and spent the remainder of the lesson completing the exercises 
from their texts or that the teacher wrote on the board. Usually, the teacher went around the 
room correcting pupils’ work and occasionally offered an explanation. Teachers seldom 
attempted to find out if individual pupils really understood what he or she (mostly she) was 
teaching. Seldom did we see pupils working together in small groups or talking with each 
other, although most SIP classrooms had pupils seated in groups. Quite a few of the teachers 
did use learning aids (manipulates) to help make a lesson more concrete - particularly in 
mathematics. We almost never saw pupils using the learning centres that SIP had helped 

teachers to develop. 
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Box 1 

Classroom Profile - Standard 3 SIP Teacher 



This teacher was using block time-tabling (teaching a class for one or 1 1/2 hours in 
one day instead of 35 minutes per day over several days) and we observed her teaching a one- 
hour English class. However, the hour-long block was not well used. She was teaching 
antonyms and would put a word on the board and ask for the opposite of that word. Most of 
the responses were group responses and at no time did she ask the pupils to use the words in a 
sentence or try to connect them to their experiences or other lessons. A lot of time was wasted 
by taking too long to call on a pupil after asking a question. When pupils wrote misspelled 
words on the board she neither corrected them nor did she explain why their spelling was 
incorrect. For example, one girl wrote “ded” instead of “dead.” This was a good 
opportunity to explain both the rule and the exceptions to the rule. Another wrote bifo for 
“before” which would have provided the opportunity to clarify the pronunciation of before. 
In 22 minutes the teacher only completed five antonyms, which would have been 
understandable if she had taken time to ensure that they really understood the words, used 
them to create their own sentences, and corrected misspelled words. - but she did not . After 
the first 22 minutes, the children spent the remaining 38 minutes working in their exercise 
books. On the board at the front of the class she had written, “Good children do not make 

noise in class. ” 

Indexes of child-centred teaching and classroom-management behaviours: The 

classroom observation instrument was designed to assess two main features of a teacher’s 
behaviour - behaviours that could be characterized as child-centred teaching, and how the 
teacher managed the classroom. For both constructs, specific behaviours were generally 
drawn from the SIP training materials. An index was created for each of these sets of 
behaviours, calculated for each teacher and summed across schools within each treatment 
group based on a weighted sum of selected behaviours. Child-centred teaching behaviours 
included: active learning in small groups; encouraging pupils to interact, help each other and 
discuss; conveying high expectations; providing pupils with learning aids; encouraging pupils 
to pose their own questions and seek answers to those questions; encouraging pupils to think; 
attending to individual students’ needs; pupils intellectually engaging with tasks, creating, 
shaping, and integrating what they are learning with what they already know, etc.) 
Classroom-management behaviours included: clearly communicating expectations for student 
behaviour; starting and stopping class on time; using time efficiently for learning; making sure 
that all pupils are paying attention and are engaged; having control over the class; having all 
materials ready and available for each lesson; lessons that flow smoothly; smooth transition 
between lessons and/or activities. 
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Child-centred teaching Table 11 shows that at the standard 3 level, teachers in all 
treatment groups exhibited substantially more child-centred teaching behaviours than did 
control teachers. Teachers in group T2 acquired 61% of the points possible compared with 
only 31% for the control teachers, and 52% for those in T3. However, the T1 teachers 
acquired only 42%, the lowest of all treatment groups. 

The range for standard 6 teachers was much narrower, with T2, T3 and control 
teachers acquiring only about 35 % of the points possible, and T1 teachers getting 44% of the 
points possible 



Table 11 

Indices of Child-centred Teaching and Classroom Management Behaviours 
Mean Percent and (Standard Deviation) By Group and Class 



Group 


Child-centered Teaching 


Classroom Management 




S3(sd) 


S6 (sd) 


S3(sd) 


S6(sd) 


T1 (4 per class) 


42(4) 


44(9) 


44(1) 


60 (5) 


T2 (4 per class) 


61 (15) 


34(7) 


88 (4) 


53 (4) 


T3 (3 per class) 


52 (15) 


39(9) 


65 (4) 


56 (5) 


Cl (4 per class) 


31 (5) 


35 (12) 


58 (4) 


65(4) 


Total Mean (30) 


46 (12) 


38(9) 


63 (4) 


59(4) 



Classroom management: At the standard 3 level teachers in group T2 exhibited a 
much higher rate of desirable classroom-management behaviours than did control teachers 
(88% vs. 58%), and teachers in T3 schools acquired 65% of the points possible but those in 
T1 schools acquired 14 percentage points less than did control teachers. 

At the standard 6 level, the control teachers scored higher than all groups of treatment 
teachers in classroom-management behaviours. SIP made far less of a difference at the 
standard 6 level in both child-centred teaching and classroom management behaviours. 

Summary: At the standard 3 level, teachers in T2 exhibited substantially more child- 
centred teaching (61%) and classroom management behaviours (88%) than any other group, 
followed by the more recently involved T3 teachers, suggesting that SIP’s work with teachers 
during Phase 2 did contribute to those teachers’ improved classroom practices. The scores for 
the T1 teachers were lower than the other two treatment groups addressed during Phase 2 and, 
in fact, the control teachers’ classroom management scores were higher than those of the T1 

teachers. 

All standard 6 teachers exhibited few child-centred teaching behaviours and there was 
little difference across the groups in either category of teaching behaviours. 
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Interestingly, these teaching behaviours are not consistent with the pattern found in the 
test results for Standard 3 teachers , where the pupils in T1 schools performed the highest, 
followed by those in T2 and the control schools, with pupils in T3 lagging behind. However, 
it is unlikely that the T3 teachers’ new teaching strategies would have had sufficient time to 
have an impact on student learning since they had only been introduced to them within the 
past 4-5 months. The regression analyses reported earlier controlled for teachers’ child- 
centred and classroom-management behaviours and found a slightly significant but negative 
influence of child-centred teaching on standard 3 test scores and no reliable influence on 
standard 6 scores. However, there were significant contributions of classroom-management 
behaviours for teachers at the standard 3 level. Graph 3 below depicts the relationships 
among child-centred teaching and classroom-management behaviours with the standard 3 
English short-answer and open-ended test scores revealing no consistent patterns except 
possibly that teachers are more adept at managing their classrooms than at orienting their 
instruction toward the needs of the children or teaching children to read and write. Graphs 4 
through 7 show the relative child-centred teaching and classroom management scores for each 
group by class level. 




Graph 4 
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Graph 5 





Graph 7 
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Box 2 

Classroom Profile - Standard 6 and Standard 3 SIP Teachers 

We observed this standard 6 teacher during a science and a business education class. 
In science, she was teaching the parts of a flower and most pupils and the teacher had a 
flower. She had written the definitions of the various parts of a flower on the board, e.g., 
“Stalk - attaches itself to the plant. ” The pupils seemed quite motivated and were attentively 
following along as the teacher explained and showed each part and then drew it on the board. 
She seemed to have good rapport with the pupils and used good questioning techniques in the 
business education class, for example, “Suppose all of us - including me - were to contribute 
100 shillings each to starting a business (someone count all of us), how much would we 
raise?" 



However, the standard 3, SIP-trained teacher at this same school had no rapport with 
the pupils. They seemed inhibited, intimidated and withdrawn. It took him a full two-minutes 
to get a response to the question “What is 5 x 3?" and when the response was incorrect, he 
said, “You are wrong. Who can help her?” 

Components of Child-Centred Teaching 

In the next section, observations of component behaviours that comprise both good 
teaching in general and child-centred teaching specifically are described. These include 
teacher questioning, how teachers responded to students’ answers, teacher feedback, use of 
lesson introductions, group learning, availability and use of textbooks, use of learning aids 
(manipulatives), classroom materials, and caring behaviour. Throughout, comparisons are 
made across treatment and control groups. 

Teacher Questioning: Observers documented the types of questions a teacher asked, 
the type of responses a teacher gave to correct or incorrect answers, and whether the question 
was asked of a boy or a girl. This was recorded for a ten-minute segment of each class session 
about five minutes after the class began. 



Questions were defined as closed or open in keeping with the definitions in the SIP 

training materials. A closed question was defined as one that has a very limited number of 
acceptable answers (usually just one) and typically elicits a one-word or short-phrase 
response. An example of a closed question is, “What is the body of a fish covered with? 
Open questions were defined in the SIP training materials as questions that, . . . anticipate a 
wider range of acceptable responses. They draw on the students’ past experience but they also 
cause students to give opinions and their reasons for their opinions. 
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Open questions were classified as open simple or open complex. An open simple 
question may haye more than one right answer, but does not particularly encourage children to 
think and analyze. An example might be, “What are sources of energy?” Open complex 
questions are intended to engage students in complex thought. For example, the following 
types of questions may require that students draw upon their own views, cause them to reason 
and analyze, have multiple possible answers, and may be open to debate: “ Why do you think 
that? How could that have been prevented? What other options are possible? How would 
you have dealt with the situation? What if . . .? What are some of the factors that may have 
led to this event? What if X had been different?” 



Most of the questions asked by teachers were closed questions requiring short, 
predictable answers and little or no analytical thinking on the part of the pupils (see Table 12). 
Teachers asked very few questions that attempted to connect to pupils’ lives or prior 
knowledge -- only about one per class session and most of these were open-simple questions 
where there can be more than one correct answer but no complex thinking is required. 
Standard 3 teachers in group T2 asked considerably more open questions than did teachers in 
any of the other groups or class levels. 



Across all observations, only about one percent of the 414 questions asked were open- 
complex questions that required the pupil to generate information or ideas and/or to engage in 
analyzing, applying, extending, or reasoning about a topic. Four of these open-complex 
questions were asked of boys and one of girls. In all categories of question types, boys were 
asked more than girls, a total of 14% more across all questions. 



Table 12 

Types of Questions Asked by Group and Class 

T1 T2 T3 C 



Totals 



Question Type/* 


S3 


S6 


S3 


cb 


25 


11 


20 


eg 


25 


5 


16 


ccnb 


2 


0 


0 


ccng 


1 


0 


0 


Oscb 


5 


5 


11 


oscg 


1 


4 


12 


osneb 


0 


0 


25 


osneg 


0 


0 


5 


ocb 


0 


3 


0 


ocg 


1 


0 


0 



* c = closed - not connect; ccn - = closed connect; osc 
oc = open complex; b = boy; g = girl. 



S6 


S3 


S6 


S3 


S6 


Total 


17 


14 


17 


27 


17 


148 


17 


13 


11 


28 


21 


136 


0 


0 


1 


0 


0 


3 


0 


0 


0 


0 


0 


1 


7 


1 


14 


2 


2 


47 


6 


1 


1 


4 


4 


33 


2 


0 


1 


4 


1 


33 


2 


0 


0 


1 


0 


8 


0 


0 


1 


0 


0 


4 


0 


0 


0 


0 


0 


1 



open simple connect; osnc = open simple not connect; 
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A high number of questions were posed to the whole class, requiring choral responses, 
with an average of 7.7 asked per ten-minute period - almost twice as many as the average 
number of closed questions asked of individual pupils. Generally these group questions 
required closed, one-word, or short, predictable responses. 

Gender differences : During the ten-minute period of observing questioning 

behaviour, boys were asked 14% more questions than girls across all question types, and four 
of the open-complex questions were asked of boys compared with only one of girls. 

Summary: Teachers’ questioning behaviour was decidedly unsupportive of child- 
centred teaching or development of childrens’ thinking and problem solving abilities. There 
was little difference in the questioning behaviour of control or treatment teachers, although the 
standard 3 teachers in group T2 did ask considerably more open-simple questions than did any 
other group. 



Box 3 

Classroom Profile - SIP Standard 3 Teacher 

This newly-trained SIP teacher was teaching English vocabulary. She used a knife as 
a learning aid to teach the words knife, blade, handle, blunt, and sharp. She had individual 
pupils come up to feel the sharp and blunt ends, but often waited too long after asking a 
question before selecting a pupil to respond. While research has found that waiting three 
seconds instead of one second after asking a question results in several cognitive benefits for 
pupils, these benefits derive only with questions that may have several responses or require 
more thinking on the part of the pupils. In this instance, the teacher was looking for only one 
correct response and her delay in calling on a pupil only served to waste valuable learning 
time. We observed several teachers waiting too long after asking a question and before 
calling on a pupil to respond. 

In this class, as with many other SIP classrooms, the pupils were seated in groups but 
did not work in groups. Her written lesson plan was also typical of most teachers we 
observed, vague and general. It consisted of statements such as: introduce by asking 

question, write the words on the board, give them work, check their work 

When the teacher began the math class, many of the pupils were not paying attention 
because they were collecting their exercise books and had to go through the stack of books to 
find their own. 
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Teachers’ Responses to Students’ Answers: The types of responses teachers gave 
to pupils’ correct and incorrect answers was documented. Categories for correct answers 
included: extends or amplifies; acknowledges as correct; praises; harsh, critical, or 

dismissive; no response; or asks other children to judge. Categories for incorrect answers 
included: requests more information; acknowledges as incorrect or incomplete; teacher 

completes or explains; teacher asks another student to answer; harsh, critical, or dismissive; 
accepts wrong answer; no response. 

By far, the most frequent response to correct answers was simply to acknowledge the 
answer as correct and move on to the next question. This was often done with a “Yes” or 
“ Good” and seldom with enthusiasm. However, when a student gave an incorrect answer, the 
teachers’ responses were more varied, including: requesting more information (26), 
acknowledging as incorrect or incomplete (28), the teacher completing or explaining (12), or 
the teacher asking another student to answer (23). 

Only six harsh (critical or dismissive) responses were documented during all ten- 
minute questioning and responding observations in all classrooms - approximately one percent 
of the teacher responses recorded. Three harsh responses were in control classrooms and two 
in SIP classrooms. Four of the harsh responses were aimed at boys and only one at a girl. 

Teacher Feedback: In 27 of the 34 classrooms, teachers gave pupils regular 

feedback, both on the responses to questions asked by the teachers, and during the latter half 
of the lesson when pupils worked in their exercise books. During that time, the teacher would 
move around the room, check pupils’ work, and enter the number correct in their booklets. 
Sometimes the teacher would ask a question of the pupil or provide an explanation of 
something the pupil did not understand. In two treatment and two control schools, the 
teachers did not provide feedback. 

Another form of feedback to pupils is displaying their work in the classroom. 
Surprisingly, only six SIP and one control teacher displayed pupils’ work. 

In addition to the teacher providing feedback to pupils, a child-centred approach 
would encourage teachers to allow or encourage pupils to provide feedback to each other. 
Only four SIP and no control classrooms showed any evidence of pupils involved in self or 
group evaluation. 
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Summary: The quality of feedback provided by teachers in most classrooms could be 
described as minimalist. Teachers seldom attempted to try to understand what children were 
thinking, to identify their misconceptions or even accurate conceptions, or to provide 
explanations that might clarify their understanding. Most of the dialogue with pupils was 
perfunctory, narrow, intellectually unengaging, and lacked a personal connection. The only 
classrooms where there was any substantive communication among pupils was in the two 
high-cost schools. In these schools, the pupils appeared to be much more comfortable and 
relaxed about interacting with each other. 

Lesson Introductions:: Lesson introductions are information provided at the 

beginning of a lesson to assist pupils in organizing the information conveyed in the remainder 
of the lesson. They are intended to assist students in ordering the subsequent material and 
help to provide clarity and integration of the material. 

Only 11 of the 34 teachers observed introduced their lessons and of those 11, only 7 
referred to a previous, related lesson, and only three attempted to connect the introduction to 
the pupils’ own experiences. Standard 3 teachers in group T1 were significantly more likely 
to use an introduction than the control teachers and standard 6 teachers seldom used an 
introduction. 



Box 4 

Classroom Profile - Standard 3 SIP Teacher 

This teacher taught two excellent lessons that exemplified child-centered teaching. 
The math and English lessons were highly integrated and the pupils were actively involved. 
The teacher was friendly and the children seemed quite happy. In the math lesson, she was 
teaching the topic of shapes. As a form of advance organizer, she asked pupils to recap the 
previous lesson on shapes by coming to her desk and picking out a particular shaped object. 
They then discussed each shape and the properties of the shape as the teacher wrote on the 
board, e.g., rectangle - two opposite sides equal: square - all sides equal. 

She then had pupils work in groups to cut out shapes. During the cutting, the pupils 
discussed what they were doing, for example, one pupil said, “No, all sides equal.” When a 
group finished one shape, they moved on to another without being told. During this time, the 
teacher moved around the room, expressing interest in each group’s work. She summarized 
the lesson by having the pupils come to the front of the room to fit cutout shapes into a large 
sheet of paper from which the shapes had been cut. 
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Even more impressive was her connection between the math and English lesson. The 
English lessori'Was on the concept of a circle and she began by having the pupils join hands to 
make a circle, then break into small groups to make several small circles. She then had them 
do other things with or about circles, such as making a circle from a rope and then make a 
sentence to describe what they had done. 

She attributed her approach to SIP training and indicated that, while she had 
acquired some of these methods at college, she had stopped using them until SIP reminded 
and retrained her. 

Group Learning: A key emphasis in the SIP training was to encourage teachers to 
involve pupils in small group learning. The observations revealed that nine of the standard 3 
treatment classrooms were seated in small groups, but none of the pupils in the control 
classrooms were seated in small groups. Groups were often large, with seven groups having 
nine pupils in them. None of the standard 6 teachers used grouping, despite the fact that SIP 
has been working with at least eight of the sample standard 6 teachers. 

Although children were seated in groups in many of the SIP classrooms, only four of 
the 22 SIP teachers had the pupils working together in their groups. This does not represent a 
significant change from that found in the evaluation of Phase 1 which reported that teachers’ 
use of individual and group work was limited. 

Summary: This may be an example of how teachers can adopt the “ trappings” of an 
idea without incorporating the essence into their teaching repertoire. Organizing pupils into 
groups is much easier than understanding how to engage them in effective learning activities, 
although it appeared that SIP did provide teachers with a number of materials and ideas for 
small group activities. Small group work is often dismissed as unworkable in developing 
countries because of large class sizes. However, many of these classrooms had reasonable 
enrolments. Those that did engage pupils in small groups ranged in size from 28 to 38. 

Box 5 

Classroom Profile - Standard 6 - SIP 

This teacher’s lesson illustrates several activities or behaviours encouraged by SIP. 
She began by asking the pupils a series of questions aimed at teaching them the use of the 
conditional tense “I would” or “I’d. ” Almost all of her questions were connected to the 
pupils’ lives and to their feelings. She began by asking, “Who has seen a rat? She then 
asked, “Where did you see a rat? In which room of your house did you see a rat?” After 
several pupils gave their responses, she asked, “What would you do if you found a rat in your 
bedroom ?. . . in the kitchen, etc, ” and wrote each pupil’s response on the board. Before class 
had begun, she had drawn several scenes on the board, such as withering crops or bountiful 
crops. After a series of questions related to the pictures that elicited responses that began 
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with, “I would, ” she then had pupils pair up and ask each other questions using the 
conditional “would” such as, “What would happen if there was no rain?” or “What would 
happen to the farmers’ crops?” 

This teacher actively engaged the pupils, connected the questions to their experiences, 
and had them work together in small groups generating and responding to each other’s 
questions. However, the questions they were assigned to ask each other were all scripted 
based on the drawings she had put on the board. Perhaps if they had been able to generate 
their own questions and answers they would be more likely to learn a larger variety of 
contexts for using the conditional tense. In addition, the teacher did not place the lesson 
within the larger context of tenses and grammatical uses, nor did she use the opportunity to 
develop pupils’ writing skills by, perhaps, having them write short passages or stories using 
the tense. 

Textbooks: In 29 of the 34 classrooms, teachers used a textbook during instruction. 
Pupils used textbooks in only about half of the 34 classes, with about 60% of treatment classes 
using textbooks, compared with only 37% of control classes. Accurate figures are not 
available for standard 6 classes since in many cases, the pupils only had with them the book 
for the class they were currently attending. 

Table 13 shows that pupils in T1 were more likely than any of the other group to have 
textbooks Only about half as many pupils in T3 had English and math books when compared 
with groups T1 and T2, and even fewer pupils had Kiswahili books (21%). The pupils in the 
Siaya schools were more likely to have fewer books than pupils in groups T1 and T2 but 
slightly more than pupils in T3 with the exception of Kiswahili books (21 vs. 12%). The 
regression analysis described earlier revealed little influence of textbooks on test scores, 
although the correlation of textbooks with mother’s education may have counteracted the 
influence of textbooks when the two variables were entered into the equation together. 



Table 13 

Percent of Standard 3 Pupils Having Textbooks by Group 



Group 



English 



Math 



Kiswahili 



T1 (4) 
T2 (4) 
T3 (3) 
C (4) 



67 (21) 
59 (38) 
34 (18) 
42 (8) 



71 (14) 
65 (33) 

35 (15) 

36 (17) 



80 (62) 
40 (29) 
21 (23) 
12 ( 7) 
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In many classes, pupils did not have their own books and shared with other pupils. 
Generally, they shared a book with one other pupil, but in eight classrooms, there were nine 
instances each in which a pupil shared a book with more than two other pupils and four 
classrooms where about half of the pupils shared with more than two other pupils. In a 
number of classrooms there were so many pupils sharing a book that several could not see the 
book. 



Learning Aids: Two-thirds of the SIP teachers used learning aids during their 

lessons, as compared with only one-quarter of the control teachers. Learning aids were more 
often used in math classes at standard 3 level. In half of the SIP classes, pupils were 
encouraged to use the learning aids. These often included bottle caps and sticks for use in 
multiplication exercises, and small pieces of cardboard with words on them for use in making 
sentences in English classes. There was remarkable similarity in the use of the learning aids 
in the SIP classrooms, likely the result of SIP influence. In no instances were boys more 
likely than girls to have access to the learning aids. 

Box 6 

Classroom Profile - Standard 3 SIP Teacher 

This teacher was also the SIP co-ordinator at her school. The first lesson was 
Kiswahili and she was teaching word use using several objects She would hold up two 
objects, such as a large and small knife and then ask “Ni up mkubwa? (Which is the big 
one?) ” “Ni up mdogo? (Which is the small one?) ” She then asked for the reasons for their 
choices. “Kwa nini utachagua kisu hiki? (Why do you choose this knife?") Children 
conversed using the words, comparing their choices and making informed preferences. 

In the math lesson, children sat in groups and used counters to make sets indicated by 
the teacher, e.g., “Three groups of two. ” She then moved to columns and had the children 
arrange bottlecaps in columns as she called out, “Show me four rows with two bottlecaps 
each. ” Although each child was working with his or her own counters, they were discussing 
with the others in their groups and cross-checking with their peers’ arrangements, suggesting 
that a culture of communication and consultation had already been established. 

This teacher also indicated that she had learned some of these methods previously but 
had stopped using them until SIP came along and reminded her of them- “Knowing 
something is one thing, but practising it is another. ” 
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Classroom Materials: SIP classrooms were far richer in materials than were control 
classrooms. This was the most distinguishing characteristic of a SIP classroom. The 
materials provided a much warmer, more interesting environment for the pupils — and for the 
observers. We received substantial feedback about the value of the materials from teachers, 
head teachers, and parents and a major concern was to find the funds to provide the non-SIP 
classrooms with materials. The control classrooms and the standard 6 classrooms that had not 
participated in SIP were bare. Usually there were no materials of any kind other than 
teachers’ and pupils’ books, desks, and a chalkboard. 

However, in a few of the SEP classrooms, the materials were faded, dirty and 
bedraggled and sometimes hanging off the wall. We never observed a teacher referring to or 
using these materials in any way, other than the learning aids described earlier. The SIP 
classrooms were also far more likely to have a clock or drawing of a clock (one that could be 
manipulated to practice telling time), a calendar, learning centres, timetables, and maps on the 
walls. 



Box 7 

Classroom Profile - Standard 6 SIP Teacher (High-cost School) 

This teacher was the only teacher we observed actually teaching the children to write, 
and her efforts were reflected in the highest scores of all tested schools in English 
composition, (most of the standard 6 classes we observed were math or science where writing 
would not be as likely to be taught.). She began by dictating some spelling words, having 
pupils collect their exercise books, and the take out their book reports. She reviewed 
incorrectly spelled words, errors of capitalization, and commented on their handwriting. She 
encouraged them to reread what they had written and to make sure that it was what they 
wanted to convey and to make corrections. There was considerable emphasis on how to 
write, but she offered mostly criticism and little praise. She then gave them a writing prompt, 
" The most interesting story I ever read, " and had them write a response. She then asked one 
girl to tell her story, then another girl. Then the teacher told a story about a girl whose 
mother was overly protective and eventually she got eaten up by a hyena. Neither observer 
quite saw the point of the story, nor did it have much engaging detail, but she did encourage 
them to read and write stories and said they were very interesting. Probably the best part of 
the lesson was that she talked about summarizing and used as examples her summaries of the 
stories the pupils had told in class. She ended the class by giving them another writing 
assignment and encouraging them to correct their spelling Interestingly, this class of pupils 
received one of the lowest Kiswahili composition scores. 
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Caring Behaviour: Observers rated teachers on the extent of their caring behaviors. 
This might have included their tone of voice, smiling, knowing pupils’ names and calling on 
them by name, or touching pupils affectionately. Only one teacher in one of the newer SIP 
schools was rated as very caring, seven SIP and two control teachers were rated as caring and 
13 SIP and four control teachers were rated as neutral. Only one control teacher was rated as 
uncaring and this teacher was from a control school. In three instances in SIP schools, the 
teacher appeared to be more caring to girls. 

Language of Instruction: Almost all instruction and pupils’ responses were in 

English, even at the standard 3 level. SIP staff indicated that parents expect teachers to teach 
in English beginning in standard 1, despite the fact that national policy is to teach in the 
pupils’ mother tongue, gradually transiting into English and teaching primarily in English in 
standard 4. English and Kiswahili are to be taught as second and third languages beginning in 
standard 1. It was not clear how much of the pupils’ failure to respond to the open-ended 
questions in the tests given for this study were due to their lack of understanding in English, 
although most pupils performed fairly well on the short-answer and multiple-choice tests that 
were administered in English. This is an important and complex issue which should be 
studied further. One of the Maseno University graduate students has submitted a proposal to 
study this issue. Perhaps AKF could work with the student and his or her advisor to ensure 
that the study addresses the issue in a technically-and pedagogically-sophisticated way. 

Gender Differences: Several items were included on the classroom observation 

schedule to detect differences in the ways boys and girls were treated. The results showed the 
following: 

■ Teachers asked about 14% more questions of boys than girls across all categories of 
question types. 

■ Boys were no more likely to get learning aids than girls. 

■ Teachers exhibited somewhat more caring behaviours to girls than boys. 

■ In only two instances did teachers exhibit any behaviour that indicated that they 
seemed to expect more of boys than girls. 

■ At the standard 3 level, there were no significant differences in scores between boys 
and girls on any test. 

■ At the standard 6 level, girls significantly outperformed boys in the English multiple- 
choice test in T1 (64% vs. 57%) and Siaya (54% vs. 48%) . 

■ A comparable number of boys and girls owned textbooks. 

Summary and Discussion 

The classroom observations revealed that, although SIP teachers practised more child- 
centred teaching behaviours than did control teachers, their use of them was limited and often 
shallow. The more important, yet difficult, behaviours such as asking questions that engage 
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pupils in thinking analytically, having pupils be active learners working in small groups, 
encouraging Children to express themselves and explore ideas, or using language to 
communicate and understand, were very rare in all classrooms. Why is this? There are hints 
that can be derived from previous research, interviews with teachers, and focus-group sessions 
with whole school staffs, each of which is discussed in the next section. 

Box 8 

Classroom Profile - Control teachers - Standards 3 and 6 

In this control school, the standard 3 teacher was teaching about the local 
administrative system. The lesson seemed too advanced for these young children. It included 
how the township administration was organized, the heads of departments, the county council, 
and the role of each of these offices. 

The standard 6 classroom was the only one observed where several of the pupils did 
not have desks and were sitting on the floor. None of the control classrooms had materials 
displayed on the walls, except perhaps a lone photograph from a magazine or an old, dusty 
chart of some sort. However, this standard 6 teacher used several desirable instructional 
strategies that she later described as having learned in her coursework at teacher training 
college, which she had recently completed. She was teaching a lesson on the volume of cubes 
and cuboids and used a cardboard box as a learning aid. She drew a cube on the board with 
the length of each side labelled. She then measured the sides of the cardboard box and 
showed that all sides were the same length. She distinguished between a cube and a cuboid 
and put the formula for each on the board and asked pupils to put the formulas in their 
exercise books and gave them several exercises to do. After they were completed, she put 
their answers on the board and reviewed them. Her lesson was coherent and well organized, 
although she did not have boxes available for the pupils to use, nor did the pupils work in 
small groups or discuss what they were doing. When she called on one pupil who refused to 
respond, she said, “ You should be sitting in the front (on the floor). She gave him a second 
chance and when there was still no response, asked, “You don’t know at all? - can’t speak? 

Sit down, ” clearly embarrassed and annoyed by the boy’s lack of response. 

What are the project teachers’ level of understanding regarding child-centred teaching 
and learning? 

Each teacher and head teacher was interviewed by one of the senior researchers and was 
asked about his or her understanding of child-centred teaching, as well as whole language 
learning, and critical thinking. In addition, the staff at each school were brought together for a 
focus-group session at which teachers were asked why their use of the child-centred behaviours 
was so limited. This section begins with a brief review of some relevant research in teacher 
conceptions and misconceptions, followed by a description of teachers’ responses to the interview 
questions and then the focus-group sessions. 
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Research on Teachers’ Misconceptions: Several studies have found that teachers or 
head teachers may misinterpret the intent of a new teaching approach or curriculum. For 
example, a recent study looked at how teachers interpreted the changes they were expected to 
make as reflected in a new mathematics framework (Cohen et al. 1990).“ The framework was 
meant as a guide for teachers and contained an interpretation of state policy to encourage and 
guide teachers to change their approach to teaching mathematics. Textbooks consistent with 
the framework were also made available to teachers. The focus of the framework was on 
developing students’ understanding of mathematical relationships, their ability to reason 
logically, to use mathematical techniques effectively, to promote attitudes of exploration and 
invention, to know both how and why skills are applied, and to develop in students a spirit of 
inquiry and intellectual curiosity toward mathematics. 

The teachers in this study were of a high caliber, admired by their colleagues, effective 
with students, eager to learn and to inspire their students, and comfortable with teaching 
mathematics. Despite all these admirable qualities, the study found that teachers’ beliefs 
about mathematics and what students should learn in mathematics were inconsistent with 
those specified in the framework. While the framework focused on helping students to see the 
inherent beauty of mathematics and to cultivate their own strategies of problem solving, 
teachers were more concrete in their goals for students. They focused on the utilitarian 
aspects of mathematics — on helping students to know how and when to use math. Teachers 
believed that they had taught something if they told the students about it and provided time to 

practice the steps. 

The authors of the study noted that, “The teachers are firm believers in the traditional 
method of direct instruction, no matter what else the framework suggests.” The method 
started with an advance organizer in the form of an example of an algorithm. Next came 
direct instruction on the procedures, making sure that each procedural step was drilled. They 
relied heavily on rules, procedures, and drill to review, followed by guided practice of text 
exercises and homework assignments consisting of additional and similar exercises. Teachers 
viewed good teaching as tracing the steps of a procedure, not tracing the student reasoning 
through a series of mathematical decisions. 

In a similar study of science teaching in Nigeria, ai Olorundare found that teachers did 
not fully understand one of the essential foundations of the new science curriculum, i.e., 
developmentalism. Areas of the curriculum they did not understand or found difficult to teach 
were simply left untaught. 
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SIP Teachers’ Understanding: In this study, we asked both SIP and control teachers 
about three of the central notions addressed by SIP: child-centred teaching and learning, 
whole language teaching, and critical thinking. However, after asking about whole language 
teaching and discovering that none of the teachers knew what it meant, we decided that, 
although SIP had offered several workshops on reading that may have been consistent with 
the whole language approach, they may not have used this terminology. However, we did not 
observe any teachers using whole language teaching strategies except perhaps in one instance 
where a teacher had pupils using word cards to form sentences. 

Below is a sample of teachers’ responses when asked if they could describe what they 
thought child-centred teaching and learning is: 

(SIP Teachers) 

■ A methodology whereby a child can be given instruction and guidance, and work with the 
materials to find things himself. The child uses nature comers and learning centres. 

■ This is a “bone of contention.” It hasn’t been fully explained. But I believe it is activity 
- and project-based teaching guided by the teacher. 

■ An approach where the child does and the teacher supervises. It involves children in 
group discussions; provides a variety of learning activities and approaches, and is based 
on the learning needs of the children. 

■ Involves children in the learning and not lecturing to them. 

■ Helps develop memory and cognition. 

■ The children enjoy learning when they are involved. 

■ Is a method where children find out things by themselves (many teachers made comments 
similar to this.) 

■ This is where most of what is learned comes from the pupils, e.g., they should ask and 
answer more questions than the teacher; should be left to work on their own and do most 
of the discoveries by themselves with the supervision of the teacher. 



(Control teachers) 

■ The child takes an active part in the learning process. 

■ The child has to be creative and teachers help the child. 

Examples of child-centred teaching : When asked to give an example of a child- 
centred strategy they use, most teachers simply responded that they now have the pupils do 
more of the talking or working in groups using teaching aids. Other responses included: 

(SIP teachers) . . 

■ In a lesson in art and craft, I give the materials and the children do the fabric decorations 

on their own. 
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■ If I want to teach about animals, children collect animals, put them in small containers, 
and classify- them. 

■ In home science I involve the children by making and eating a dish of food together. 

■ I break big topics into sub-topics. 

■ In teaching about the rainbow, I make the children make foam from Omo (a powdered 
detergent) and water and see the colours forming. Then I name the colours. This ensures 
they observe the colours even when the rainbow is not there. 

■ Now we have many materials and if I’m sick, children can just take cards from the 
learning centre and do the work. 

Interestingly, the control teachers’ responses were quite similar. Their question was 
modified to ask “ Do you think your teaching has changed to become more child-centred in 
the past 3 years? Can you give an example of how it has changed?” 

■ I have improved the use of teaching and learning aids and most of the lessons I plan are 
practical. 

■ Children sit in groups and come up with ideas. In a lesson about birds, the children 
collect birds. 

- Counting their own teeth. (This teacher had just taught a lesson on carnivores and 
omnivores and had the children count their teeth in relation to understanding carnivores.) 

■ I refer to the previous lesson and connect information with the lesson. 

Critical Thinking: Teachers were also asked how they help their pupils develop 
critical thinking skills. Although the term critical thinking may not have been used in the SIP 
training, the interviewers used several terms to convey the more general notion of developing 
higher levels of thinking ability. Although some of the head teachers were able to offer 
reasonable descriptions of critical thinking, the teachers’ responses were vague: 

■ By asking them questions. 

■ By asking difficult questions. 

■ By giving them work to do on their own. 

■ Giving them pictures and asking them to find out what is going on in the pictures. 

■ Ask questions why, where, how, etc. 

■ By telling stories 

■ Involving them in discussions where they think for themselves. 

■ Asking them to solve problems by evaluation. 

■ We need more training in questioning strategies. “ We were taught that our questions 
should involve reasoning and analysis but we don’t know how to tell what reasoning or 

thinking is.” 
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Inadequate conceptions of child-centred teaching: Although most of these descriptions of 
child-centred teaching touch on an element of it, most are incomplete and suggest that teachers do 
not fully understand the concept. For example, several teachers responded that it is a method 
where children find things out by themselves. . . “Children do most of the discoveries by 
themselves with the supervision of the teacher.” Children can learn on their own.” Children 
work when the teacher is absent.” Did these teachers mean that children develop a deeper 
understanding by engaging in relevant activities that allow them opportunities to discover 
knowledge? Or, do they literally mean that a teacher’s role is diminished and minimal? - that he 
or she simply sets out materials and the children do most of the learning on their own. If the 
second interpretation is close to what they believe child-centred teaching to be, it may help to 
explain why they were reluctant to use it. It is not likely that they could accept that schooling 
involves leaving children to learn on their own without planning, organizing, and structuring by 
the teacher. If teachers believe that child-centred teaching means a “ hands-off” approach, then 
this could be incompatible with their sense of responsibility as teachers and cause them to 
abandon or substantially adapt the child-centred strategies promoted by SIP. 

Cohen’s research iv found that teachers must believe that the changes they are expected to 
make will result in a better situation for their pupils. Some innovations that may be intellectually 
appealing to teachers may be difficult to implement or inconsistent with their instructional 
repertoire.’ This may cause teachers to perceive them as a threat to their students’ success and so 
be resisted. In most African countries, students’ and teachers’ success is judged by students’ 
scores on national examinations. End-of-primary exams determine which students are allowed to 
continue to secondary school, and the pressure of the exam exerts substantial influence on 
teachers’ practice. 

Orungbem found that Nigerian teachers neglected the child-centered methods advocated 
in the national social studies curriculum in order to teach to the more factually-oriented national 
examinations. And Alao and Gallagher found that Nigerian chemistry teachers used the exam 
study guide to determine the content of their lessons. In Botswana, Rowell and Prophet’" found 
that teachers substantially reduced the number of hands-on science activities in order to give 
students time to complete worksheets aimed at preparing for the exams. 

Perceived Incompatibility with Curriculum : Teachers’ comments gathered during the focus 
group sessions helped to clarify teachers’ dissonance. Once we began seeing that many of the 
SIP-trained teachers were not using small groups or engaging the pupils in discussions, we began 
reporting this and asking, “Why, when you all seem so enthusiastic about SIP and child-centred 
strategies, aren’t you using these strategies more.” Teachers’ responses were honest and 
revealing and help explain the lack of child-centred teaching we observed in the classrooms. 
“Their (SIP) methods are time consuming. “We don’t do what SIP wants because we need to 
cover the syllabus and we only have 35-minute periods. In lower-primary (they use timetabling 
and so) have one-hour blocks (but we don’t have that at the upper primary level). “If they cut 
back on the amount of material covered in the curriculum, we would have more time to engage 
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the children in thinking, but now we avoid asking questions that might require any extra time. 
Some of the topics are too detailed for primary level and many topics are repeated in several 
subjects.” 

POs’ explanations confirmed what we were hearing from school staffs. We asked the 
POs, “ Why are we not seeing teachers behaving in ways that are more consistent with what 
SIP has been promoting?” They responded, “There are several reasons. First, the school 
examinations work against child-centred methods. Examinations are a high priority in Kenya 
- perhaps the highest priority when it comes to schooling, and teachers do what they think will 
cause their children to get higher scores on the exams. I think it’s difficult for them to see the 
direct link between child-centred methods and increased test scores.” 

“ Another reason is that child-centred approaches require more work for the teacher - 
at least until they have established a repertoire of materials and behaviours and a level of 
comfort with the new strategies. Often, teachers complain that these strategies are just too 
much work. Teacher and head teacher transfers are another problem. There are a tremendous 
number of transfers in the Municipality. The curriculum is also a problem. Teachers have 
only 35 minutes per core subject per day. This can make it difficult to do activity -based 
lessons, ^^e have tried to show them how to do block schedules and some schools are using 

it.” 



This may be one of the most powerful explanations for teachers’ behaviours. The Kenyan 
curriculum is very full and the end-of-cycle exams are very comprehensive, factually oriented, 
and not particularly child-centred. This has been an issue of debate throughout Kenya for quite 
some time and is not likely to be resolved by SIP. Moreover, although there is a lot in the 
curriculum, we could not find a coherent approach in either the textbooks or the curriculum that 
would induce teachers to teach reading and writing. There is no subject called reading or even 
language arts. The English texts teach grammar almost exclusively. Children do have readers, 
but the stories were dull and uninspired and the teachers did not seem to know how to use them to 
help children actually understand and make meaning out of what they were reading. This is a 
serious deficiency in the Kenyan curriculum which perhaps SIP could address at the national 
level, especially with the assistance of the International Reading Association’s international 
volunteer service. However, reducing the volume of the curriculum or changing the exams may 
prove intractable. 



SIP’s assistance with curriculum management: SIP worked with teachers and head 
teachers in a variety of ways to help them to manage the curriculum. We asked their views 
regarding SIP’s support in this area. 

Has SIP been helpful to you in managing the curriculum and, if so, how? 
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Head teachers: Head teachers’ responses revealed some dissonance about the 
curriculum. Over half of the head teachers felt that there is too much material in the 
curriculum for the pupils to understand and that some of it should be removed but all (both 
treatment and control heads) felt that the pupils can learn all of the material in the curriculum 
if the school staffs do their jobs well. Most of the head teachers in the SEP schools indicated 
that the SEP project had shown them how to manage the curriculum or to help their teachers 
manage it. 

■ I am now in a position to know what the teacher is handling. I used to make only one 
scheme of work for all years but now I make a scheme each term. 

■ The activity methods they taught us really take time. We do the curriculum in a 
concentrated way. 

■ Improved teaching methods facilitate student understanding so a teacher doesn’t have 
to keep teaching a topic over and over. 

Teachers: Teachers’ responses were quite varied on this issue and many teachers 
sounded overwhelmed by the amount of material in the curriculum. However, when 
compared with the control teachers’ responses, SIP teachers seemed to feel more confident 
about managing the curriculum. We asked, “Please describe how you cope with the 
curriculum.” 

■ Through remedial teaching in the evening. 

■ We have extra tuition (tutoring) at games time in the evening; in the mornings before class 
starts, or during class time if some teachers are absent. (Presumably some of the teachers 
charge for this.) 

■ SIP POs have helped us to see that there are topics that cut across several subjects and how 
to cluster the overlaps. 

■ Teachers do not teach all the topics in a year 

■ I am trying my level best to cope but it is too heavy for me. (Several responses were 
similar to this) 

■ Coming to school daily and teaching regularly; extra time should be available to give fast 
pupils extra work and to spend time with slower learners. 

■ By giving extra work and working harder. 

■ They come on Saturday to do remedial teaching, or early (one way to cope with the 
curriculum because of too many games, athletics, etc.) 

■ If you teach well and avoid absenteeism and laxity, you can cover the curriculum. 
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Inconsistent with several of the statements above, almost all of the SIP teachers 
disagreed withthe statement that, “ It is impossible to teach it (the curriculum) all well.” And 
almost all agreed that, “There is a lot of material to cover, but the SIP project has shown us 
how to manage it.” Most of the control teachers felt that, “The pupils cannot leam all of the 
material contained in the curriculum even if they study hard.” but only about 20% of the SIP 
teachers agreed with this statement. 

When asked to indicate the three most important factors that influence how much their 
students leam, the SIP teachers overwhelmingly selected, “How well I teach,” (16 of 22), 
textbooks (17 of 22), and the curriculum (8 of 22). Control teachers were more likely to select 
textbooks (5 of 8) and the nursery school the children attended (4 of 8). In addition, SIP 
teachers overwhelmingly indicated that they believe their children are hardworking, willing to 
leam, and enjoy activity methods. None felt that the students are lazy and only three felt that 
some of the children are unteachable. SIP teachers are consistent in their acceptance of their 
responsibility in influencing student learning. 

Box 9 

Classroom Profile - Standard 6 teacher - not SIP trained 

This lesson was perhaps the greatest waste of time of all those we observed. He wrote 
on the chalkboard a price list of items one might find in a store and then began by having the 
pupils tell the price of each item on the list. He then wrote a problem on the board and asked 
the pupils to say how many items were on the bill and wrote each item on the list again. He 
then drew a rectangle intended to represent an invoice for goods bought and sold and 
proceeded to rewrite each of the items that were on the price list onto the bill - a third time. 

How Effective was the Project’s Implementation? 

The project objectives for Phase 2 are based in large part on the evaluation findings of 
Phase 1 described earlier. Specifically, the objectives for Phase 2 were to: extend the project 
to an additional 27 schools at the lower primary level and to nine of these schools at the upper 
primary level; integrally involve head teachers in the training and improvement program; 
continue making communities and parents aware of the program activities and motivate 
parents to contribute toward the cost of the teaching materials; improve the cost-effectiveness 
of the program; and create a better understanding of child-centered learning. Strangely, the 
proposal said nothing about addressing the school as a whole, despite that being a prominent 
recommendation in the Phase 1 evaluation. 

SIP’s progress toward achieving each of these objectives is discussed below but is 
preceded by a brief summary of their accomplishments and a discussion of their work from 
the perspective of a systemic view of school reform. 
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Summary of accomplishments: SIP accomplished a considerable amount during 
Phase 2. They, did work with lower primary teachers in 27 schools and upper primary teachers 
in 13 of these schools -- four more than specified in the proposal; they provided instructional 
materials to over 100 classrooms and teachers and trained teachers in their use; quite late in 
the project they began working with head teachers but worked with School Co-ordinators 
from early on; they met with the parents and School Committees to introduce them to the SIP 
model, to enjoin their participation and support and to encourage contributions toward the cost 
of the teaching materials, desks, windows, locks, and doors; they established a close and 
productive relationship with the MEO and the Municipal Council resulting, in part, in 
replacement and training of many of the inspectors and TAC tutors; they established working 
relationships with institutes of higher education, including establishing an agenda for research 
on school quality; and instilled an ethos of school improvement and attention to the concepts 
of child-centred teaching throughout the Kisumu Municipality. 

A systemic approach to reform: This broad range of activities collectively 

represents a systems view of school reform that acknowledges the complex and 
interdependent relationships among the various parts of a system and attempts to identify 
points of leverage in the system to induce change. In a recent review of research and trends in 
the professional development of teachers, Sparks (1993) found that systems thinking was one 
of the three ideas that had influenced all facets of education reform, including curriculum, 
textbooks, testing, and teacher training — the other two being results-driven education and 
constructivism.™ 1 Making improvements that result in changes for all students requires 
coherent learning for all school staff and develops the capacity of the school as an 
organization to solve problems and renew itself. Sparks’ review concluded that the entire staff 
should be assisted to pursue incremental annual improvement related to a set of common 
objectives, such as helping all students to become fluent and engaged readers. 

Objective 1: Extend the Project to an Additional 27 Schools at the Lower Primary Level 
and to Nine at the Upper Primary Level 

SIP did meet this objective in that they worked with individual teachers at the required 
number of schools and, even exceeded the objectives by working with upper primary teachers 
in 13 schools. However, although SIP addressed the broader educational system in Kisumu, 
they did not adequately address the school as a system , focusing instead on individual 

teachers. 

In judging SIP’s effectiveness in working with schools, we first discuss the evidence 
regarding SIP’s work with the school as a whole and then draw upon the various elements of 
effective professional development programs revealed in Sparks’ recent review of research to 
estimate the quality of their work in staff development. 
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SIP’s Influence on the School as a Whole: We fully expected to see evidence of 
focus on the whole school during our site visits to schools and were surprised at how little we 
saw. During the focus group sessions, we asked if SIP had had an impact on the school as a 
whole. Most staff indicated that SIP had not made an impact on the entire school, but a few 
reported that sometimes the SIP POs would talk to, or train, the entire school staff when they 
identified an area in which the school needed to improve. Others mentioned that the School 
Co-ordinator often shared what he or she had learned in SIP workshops with the whole staff. 
In some schools, those that attended training returned to their school and trained the rest of the 
staff. In some instances, what they had learned in the SIP workshops impacted other teachers 
through the subject panels. At one school, they referred to these teachers as “agents of 
SIPAK.” Several expressed that they hoped that SIP would devote more resources to the 
upper primary level and would work with the whole school rather than just at the lower- 
primary level. 

However, SIP may have reached beyond individual teachers in some cases. Several of 
the staffs reported that they talked more about teaching and learning with each other because 
of SIP. In fact one T2 staff said, “Yes, we talk very often - everywhere - even when we’re 
going home. We teach each other.” Another reported, “When SIP came, it came with a lot of 
work, so we have to sit down and talk about it.” A new SIP school reported that non-SIP 
teachers go to SIP teachers for materials and suggestions. At one school, the staff noted that, 
“Just the anticipation of a SIP visit would make the teachers work harder. They suggested 
that one way that SIP could have a greater impact would be to have more frequent visits to the 
schools. 



Skill Mastery: Sparks’ review revealed that effective staff development efforts are 
ongoing and coherent and based on a clear, compelling vision of how the school system 
should look. He found that fragmented and piecemeal staff development has contributed to 
poorly understood innovations with teachers unable to master a new skill before they are 
expected to move on to the next area of reform. Although SIP s training efforts should not be 
characterized as fragmented and piecemeal, the fact that most of the teachers seldom exhibited 
the most central child-centred teaching behaviours during our observations (e.g., having 
students work together in small groups and discuss with one another), suggests that SIP may 
have moved too fast with teachers in their race to meet the project goals of working in 27 
schools. 



In the conversations with the POs, they confirmed this view: “ The schedule we have 
set up to work with schools may be unrealistic. We typically work with 3-to-4 schools per 
academic term (four months). We would prefer to work with a school for one year in teams of 
two POs working with two schools per year. In four months, teachers are not able to fully 
integrate the new behaviours and as soon as we are not there to support and coach them, they 
fall back into their familiar ways. 
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SIP had a staff person assigned to conduct research and monitoring studies. It would 
have been useful for that person to study whether teachers were indeed integrating and 
retaining the new skills they were learning, and, if not, figure out what it might take to ensure 
sustained behaviour change. 

Focus on Students’ Needs: Another finding in Sparks’ review is that instead of 
designing staff development to address teachers’ needs, their needs should be situated within a 
larger context that focuses on addressing students’ learning needs. SIP did focus on the 
importance of meeting students’ needs and this priority was expressed occasionally during 
interviews with teachers and head teachers. “ I now know that it’s important to be close to the 
childrens’ needs and problems and that it is helpful to have statistics to evaluate pupils’ 
progress.” However, we seldom observed teachers working with individual students in a way 
that would suggest that they were attending to individual needs. So although SIP staff may 
have promoted this idea, they did not appear to have solidly entrenched the application of it 
into teachers’ classroom practice. 

Use Multiple Forms of Teacher Learning: Sparks’ also found that multiple forms 
of teacher learning are viewed as more effective than the traditional away-from-school 
workshops. Strategies found to be effective involve teachers in studying their craft rather than 
being handed expert knowledge from above. Other effective strategies include teachers 
conducting action research, participating in study groups, solving problems in small groups, 
observing peers, keeping journals, and becoming involved in the improvement processes. 
Similarly, Wheeler et al.’s (1989) review of in-service policy in Thailand found “one-shot 
training courses” to be ineffective, u and a study of in-service training in Indonesia found that 
intensively trained teachers made more substantial changes and had greater student 
achievement than did those who received minimal training (Anderson and Djalil, 1989.) x 
Raudenbush et al.’s (1993) xi review reached similar conclusions: 

The literature seems to indicate that short-term courses without classroom follow-up 
are unpromising. Effective in-service instruction apparently requires classroom 
demonstrations, opportunities for teachers to practice and refine pedagogical 
techniques, and sustained follow-up, supported by classroom observation and 
feedback. Teacher involvement in the identification of course content and materials 
may also be important. The effective models appear to be intensive and, therefore, 
expensive. 
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SIP certainly avoided the error of providing one-shot courses and did provide 
classroom demonstrations, opportunities for teachers to practise and refine their techniques, 
and sustained follow-up and support, although whether it was sufficiently sustained is a 
central question. However, SIP did not involve teachers in studying their craft. The model 
used was to have an expert (or novice-expert in the case of the SIP POs) conduct a workshop, 
have the participants discuss what the expert was conveying in the workshop setting, and then 
go back to their classrooms to try to implement what they learned -- often with the assistance 

of the PO. 



However, it may have been more compelling to have had school staff meet regularly 
as a whole (perhaps with other staff from a neighbouring school), and have the POs serve as 
facilitators of craft study. This might include working with the staff to identify problems that 
they want to work on for a period of time, such as poor reading comprehension, giving the 
staff reading material such as research summaries written for practitioners, perhaps showing 
videos of teachers exhibiting desirable behaviours in developing childrens’ reading abilities, 
discussing what they think of the research and videos and how they might try these strategies 
out in their own settings. The POs could facilitate these discussions as well as teacher 
exchanges to observe and assist each other in practising the new strategies or discussing them 

in follow-up sessions. 

As far as we could discern, SIP also did not take advantage of these other research- 
based effective strategies such as having teachers conduct action research, participate in study 
groups, observe peers, or work in small groups to solve problems. Each of these strategies is 
aimed at imbuing the staff of a school to establish a clear vision of quality schooling and to 
work together as a team to study, analyze, and solve their own problems - much the same as 
children would do in a truly child-centred classroom setting. Kisumu school staff were not 
left with the skills and attitudes to be their own problem solvers, although they were more 
sophisticated about and receptive to what it takes to ensure quality teaching and learning. 

Sparks’ review also found major changes in the role of staff developers. They now 
provide consultation, planning and facilitation services in addition to conducting training. For 
example, they may facilitate meetings held by a school to address specific problems or 
develop long-range plans. This may mean that the SIP POs need to develop their own skills in 
systems thinking, results-driven education, meeting facilitation, long-range planning and 

consultation. 
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Teacher Networking: In Ingram’s study of the school change process, he found that 
the most important feature of a school’s improvement was teachers’ interaction with other 
teachers. When teachers worked together, they formed a critical mass that was able to 
overcome obstacles and provide emotional support. This was supported in Orungbemi’s study 
of curriculum change in Nigeria. He found that over 70% of the teachers and head teachers 
felt that a lack of professional exchanges affected their ability to successfully implement the 
curriculum. And Adams’ study of teachers’ efforts to implement a new mathematics 
curriculum found that a network approach to staff development had a much greater impact on 
teacher practice than did traditional staff development or a support-group approach/'' 

The teacher networks described by Adams extended beyond the support-group 
approach where teachers met on a monthly basis to develop curriculum materials, share 
experiences and frustrations, and further their understanding of the curriculum innovation. 
The network approach added a common within-school preparation period for the teachers, 
cross-school monthly meetings, monthly staff development workshops that linked the 
“ treatment” teachers with non-treatment teachers, intensive summer training, and a “ linker” - 
- a professional whose responsibilities included connecting teachers to an extended body of 
professional expertise. A critical feature of the network model was the links with professional 
organizations and resources beyond the teachers’ district. 

SIP did little to promote teacher networking beyond the times that teachers interacted 
in the workshop sessions or through meetings held at the TACs. Finding additional ways to 
implement and institutionalize the networking process, perhaps through nurturing professional 
teacher associations in the subject areas could help to ensure that responsibility for 
professional growth is disbursed throughout the educational community in ICisumu. 



Summary 

SIP did meet and even exceeded the first objective of working with 27 lower primary 
schools and 9 upper primary schools. However, they did not focus on working with the 
schools as a whole. Moreover, in their work in training teachers, they did not appear to bring 
teachers to a sufficient level of skill mastery in using child-centered techniques, in part 
because they moved from one school to the next before the new techniques and philosophy 
had been fully integrated. Although SIP did promote an ongoing and somewhat coherent 
professional development agenda supported by classroom based work with teachers, they did 
not take advantage of several other forms of teacher learning that have been shown to be 
effective, such as teachers studying their craft or teachers networking with organizations and 
resources beyond the their district. 
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Objective 2: Integrally Involve Head Teachers in the Training and Improvement 

Programmme.,. . 



Research on Head Teachers: The role of the head teacher consistently has been 
found to be central to the success of an effective school. If a head teacher disapproves of or 
dislikes an innovation, success is unlikely, but if he or she provides guidance and moral 
support to teachers, change is favoured. For example, a recent study of factors associated with 
the most effective schools in Pakistan™ found the two most important contributors to 
successful schools (i.e., those that steadily improved their educational quality) were the 
presence of a strong head teacher and a vigilant and supportive community. 

The head teachers in these successful schools fostered collaborative relations among 
the school staff, which in turn resulted in co-operation and experimentation. Teachers 
encouraged and assisted each other and this co-operation enabled them to continuously 
improve their teaching methods. In contrast, there was limited contact and sometimes conflict 
among teachers in the control schools. While all the schools encountered similar problems, 
the successful schools, aided by the participatory environment fostered by a strong head 
teacher, were able to overcome problems through active problem-solving, both inside the 
school and with the community. The poorly functioning schools often took a shallow 
approach to problem-solving and often at the initiation of those outside the school. The 
successful schools also had a sense of shared goals among the teachers, students, and 
community. 

Raudenbush et al. xiv also found head teacher supervision of teachers to be the most cost 
effective of three approaches to in-service teacher training in rural primary schools in 
Thailand. The approaches compared were: in-service training courses, external supervision by 
district or circuit supervisors, or internal supervision by either head teachers or teacher leaders 
based at the school. Internal supervision was as effective as pre-service education but external 
supervision had no effect on student test scores. The advantages of internal supervision are its 
regularity and the fact that it is based on direct observations of teaching behaviour. External 
supervision lacks the regularity of internal supervision. 



The critical component in this study was estimated to be the head teacher’s ability to 
create and sustain an academic focus and an “ethos of improvement’ which, for example, 
encouraged teachers to come to school on time, to use test results to evaluate instruction, and 
to discuss teaching and learning during lunch breaks. The more effective head teachers also 
were successful at motivating the community to provide resources to purchase instructional 
materials and at identifying district level resources. Internal supervision also has obvious cost 
benefits since the supervision is conducted by those already employed and at the school. 
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SIP’s Work with Head Teachers: The Phase 1 evaluation report expressed a 

concern that SIP had not provided sufficient attention to the school as a whole or to the role of 
the head teacher in its school improvement efforts. Despite this concern and recommendation, 
SIP did not begin training head teachers until late in 1994, 18 months into Phase 2. In 
interviews with teachers, head teachers and school staff as a whole, all agreed that the head 
teacher did not serve as an instructional leader, seldom visited teachers’ classrooms, and, in 
fact, did not even see it was a role they were supposed to be playing. 

Head teachers may have attended SIP or other workshops offered to teachers or open 
to education staff within the Municipality before late in 1994, but no deliberate, systematic 
effort was aimed at the head teachers in Kisumu until then. Once SIP began offering the 
training, SIP head teachers seemed to take advantage of it. Those interviewed reported 
attending substantially more workshops (either SIP, SPRED, or other non-SIP workshops) 
than did control head teachers. Head teachers in T1 attended an average of 22 workshops, in 
T2 - 18 workshops, and in T3 - 17 workshops, but in control schools only 3 workshops. 
(These were not necessarily separate workshop sessions, but rather different workshop topics 
that may have been addressed within a single session.) However, when the number of 
workshops attended by head teachers was entered into the regression equations, they did not 
seem to have an influence on students’ test scores as did teachers’ workshop attendance. This 
may be because a number of the workshops for head teachers were aimed at administrative 
and fiscal issues. It was only recently that SIP began working with head teachers on 
instructional leadership. 

School Co-ordinators: SIP did institute the role of the School Co-ordinator in the 
SIP schools and they did serve in some ways as instructional leaders. Co-ordinators were 
teachers in the target schools and were selected by the school’s head teacher They served as 
resident experts working alongside their colleagues and received additional training to help 
them assist teachers in implementing SIP activities once the SIP team moved on to other 

schools. 



SIP head teachers reported that the School Co-ordinators had been useful in serving as 
a link between teachers and the SIP staff, sharing what they had learned in the SIP workshops 
with teachers, and training staff in what the Co-ordinator had learned in the workshops. 
Several head teachers reported using the School Co-ordinator as an assistant to the head 
teacher in working with teachers. One of the T2 head teachers, however, reported that the Co- 
ordinators (there were two at that school) had been helpful until he (the head teacher) had 
received SIP training himself, suggesting that the head teacher then felt empowered to assist, 
guide and train teachers himself. However, two of the head teachers in T1 reported either not 
using the Co-ordinator or that the Co-ordinator had not done much, in contrast with the 
teachers at those same schools who reported that the Co-ordinator had helped teachers to 
implement the curriculum, make teaching aids, and provided materials and teaching advice. 
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Half of the SIP teachers responded that the co-ordinator’s role was an important one 
and should be'maintained when SIP activities are absorbed into the MEO. However, during 
the focus group sessions with the entire school staffs, several teachers reported that they were 
not sure what the School Co-ordinator’s role was and suggested that it should be clarified. 
One T3 teacher reported that the co-ordinator encouraged teachers to co-operate with each 
other so as to work in a good atmosphere and that she (the Co-ordinator) keeps checking to 
make sure that teachers are in class. “Some people need to be reminded.” Another 
acknowledged that, “ The Co-ordinator understands our problems much better than the head 
teacher” and yet another said that the co-ordinator encourages teachers to use teaching aids 
and answers the teachers’ questions promptly. One of the Co-ordinators interviewed reported 
that he felt that he improved as a teacher because of the opportunity to supervise other 
teachers. 

What Did the Head Teachers Think of SIP? During the interviews with head 
teachers, they were asked several questions about SIP’s influence on their views of teaching 
and learning and their practice as head teachers. Their responses indicated considerable 
support and enthusiasm for SIP and its message. For example: 

■ Teaching has not changed much in the upper classes, but SIP has changed me very much. 
I now try not to dominate the lesson and I sit down while the teachers teach me. I also use 
a variety of instructional strategies now. 

■ lam better able to have a free discussion with the teachers and to guide them. I have 
adopted a few methods that have helped me personally as a teacher (e.g., child-centred 
methods, questioning). The KCPE results have risen from 260 in 1988 to 410 in 1994. 

■ School administration has become easy. 

■ It has enhanced my abilities in leadership. 

■ I’ve learned that these learning centres, work cards, and child-centred methods can be 
extended to children in standard 8. 

■ lam now able to check teachers’ lesson plans and schemes of work. 

■ I now know how to approach teachers and guide them without quarrelling. 

■ There is an open and frank atmosphere of discussion with the teachers 

What Did Teachers Think of Head Teachers? Each teacher was asked to indicate if 
there had been any change in the quality of guidance provided by head teachers, and if so, 
how the guidance and support had improved. Half of the SIP teachers interviewed indicated 
that the guidance from their head teacher had improved since SIP began. However, almost 
half of the control teachers also felt that their head teacher’s guidance had improved over the 
past three years - also by a lot. Teachers’ comments included: 

■ He comes to class, observes a lesson and positively points out areas that need 
improvement. 
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■ He has asked parents for money to buy more materials and encourages staff to use the 
materials. .. 

■ He encourages me to use child-centred methods. 

■ He gives advice to the teacher where necessary. 

■ He informs us of SIP courses and occasionally provides transport for courses. He 
organized a SIP course for the whole staff. 

■ He encourages me to work harder. 

■ He reminds me to use the methods of teaching required in all subjects. I find it easy and 
enjoyable. 

■ He’s now more interested in checking whether you have used the materials given and 
whether you have displayed what you have made with them. He wants classes to be full 
and with comers of interest. He motivates the teachers. 

■ When the head teacher visits the class regularly, the children become motivated and 
alert. The teacher is happy and teaches better. 

Inspectors and TAC Tutors: SIP also had responsibilities to work with the school 
inspectors and TAC tutors. When asked if there had been any improvement in the guidance 
from inspectors since SIP had begun, only seven of the 22 SIP teachers indicated that the 
guidance had improved. Some of their comments on inspector guidance were: 

■ He gives advice on grouping and use of learning aids. 

■ He is discouraging and negative (another teacher from the above school). 

■ He assists with helping to make the schemes of work and lesson plans more efficient. 

When the control schools were asked about the inspectors, one staff reported that, 
“ The last time they came, the staff felt disappointed. There was a lot of harassment. A few of 
the teachers were not prepared and inspectors capitalized on that. They leave teachers with a 
scar - they’re too rigid. They should come as advisors - not fault finders.” 

Several of the head teachers indicated that the workshops offered by the TAC centres 
are more relevant and practical since SIP’s involvement, and that they are more likely to be 
based on having identified teachers’ needs than previous workshops. During the last year of 
SIP, several of the TAC tutors were replaced at SIP’s urging and SIP began working more 
closely with the tutors to include them in training, technical assistance, and planning activities. 
However, only five of the 22 SIP teachers indicated that the guidance from the TAC tutors has 
improved since SIP began. 

■ The number and frequency of workshops has improved. 

■ He helps the teacher to make better teaching aids (several comments). 

■ She assists with teaching resources. 

■ He makes frequent visits to the school. 
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A notable feature from our observations and interviews is the fact that all but one of the 
head teachers at the schools in the study sample were male, as were most of the TAC tutors 
and inspectors. However, about half of the SIP staff were female. 

Summary 

The intent stated in the project proposal was that head teachers would “ take charge of the 
ongoing teacher training on completion of the project and” . . . with the TAC tutors would 
“ train the other teachers in the school.” It is unlikely that the head teachers in Kisumu are 
sufficiently equipped to assume this responsibility, primarily because SIP began working with 
them late in the project and because much of the training they did receive addressed 
administrative and fiscal issues rather than instructional issues. 

The head teachers we interviewed seemed to be very receptive to and appreciative of the 
new ideas and skills they were learning through the SEP workshops and contact; some were 
even beginning to exhibit behaviours of effective school leaders. 

Although SIP did incorporate the School Co-ordinators into the schools to support 
teachers’ ongoing development, it is disappointing that SEP waited until almost five years into 
the project to begin training head teachers to be instructional leaders, especially with the 
abundance of research pointing to the central role of the head teacher in school and teacher 
effectiveness. It is quite likely that SIP’s impact would have been much greater if it had 
involved head teachers from early on. The continuation and sustainability of positive change 
in schools depends on the presence of a strong head teacher, a committed teaching staff who 
work as a team, and the monitoring and support of the community. 

In addition, changes are sustained when those who undertake the changes remain in the 
school so that the changes become routinized. Transfers of school staff in Kisumu were 
rampant, but it was unclear who was responsible for the transfers. This is another factor that 
will need to be addressed if SIP’s efforts are to be sustained over the long term. 

To SIP’s credit, it made a concerted effort to upgrade the quality of the Municipal staff 
whose responsibilities are to support educational improvement. They did this by encouraging 
the MEO to replace ineffective staff and by working closely with the inspectors and TAC 
tutors to introduce them to SIP’s views and rationale for school improvement. 
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Objective 3: Continue Making Communities and Parents Aware of the Programme 
Activities and Motivate Parents to Contribute Toward the Cost of the Teaching Materials. 

School Committees are groups of parents and community members from the school’s 
locale who are responsible for the fiscal oversight of the school. In the Kisumu Municipality, 
School Committees and PTAs have been combined to form one organizing body. School 
Committees meet at least once-a-term (three-times-a-year) and, based on the interviews, are 
responsible for the following issues: providing physical facilities for the school such as land, 
buildings, desks, water, and electricity; providing labour for building facilities, raising funds, 
maintaining a school budget; overseeing the expenditure of school funds or funds raised for 
building projects; serving as a link between the school, parents, community, MEO, and 
Ministry of Education; providing learning materials such as textbooks and library books; and 
ensuring teacher and pupil discipline. One School Committee also reported that they had 
established a burial and benevolent fund for pupils, teachers, and parents in response to the 
recent AIDS crisis in the community. 

SIP approached each School Committee prior to involving the school in SIP and at 
those meetings specified the role and responsibility of the Committee in providing secure 
classrooms, etc. in order for the school to become a SIP school. A few of the Committee 
members reported that they had attended SIP training and found it very useful and inspiring. 
They particularly liked the sessions on school management, bookkeeping, budgeting, the 
relationship between the Committee and the school’s administration, and the role of the 
School Committee. Most requested more training from SIP. When asked if the training they 
received from SIP has led to improvements in the school, responses included. 



■ Yes, we’ve added more desks and tables to the classrooms. 

■ The SIP levy for materials has been initiated. 

■ The School Committee has agreed to sustain the SIP project. 

■ We provided the materials needed to make the classrooms secure. 

■ We began our participation with SIP in February 1996 and since then have constructed 
wire mesh windows and added three doors. Plastering of the floors is in progress. Pit 
latrines have been dug and we are planning to build furniture for the head teacher s 
office, cupboards for books, chairs, and tables. 

Committee members were asked if the community’s view of teachers in their school has 
changed since SIP began, and if so, to describe the changes. Responses included; 

■ We now like teachers because they now teach better and there is more co-operation 
between them and the community. 

- Since this project started, the standard of education in this school has greatly improved. 
Our children can now read and write. Now they do their homework at home. We are 
very happy with our teachers because they sacrifice their free time to teach our children. 
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We have always had positive attitudes towards teachers and our views have not changed. 
Teachers are now responsible, they come to school regularly. 



During the focus group sessions with school staff, we asked teachers what influence 
they thought SIP had on parent and community involvement. The responses were quite 
inconsistent across schools. One school reported that the materials provided by SIP have 
influenced more parents to want to enrol their children in that school and to contribute money 
to buy materials for their own child’s classroom. However, another said that parents are 
staying away from the school because they think that the school will ask them for money or 
that their children have done something wrong. “ Many have not even come to collect last 
term’s grades.” Another said that, “The parents were ready to pay to buy the materials when 
they heard that SIP was winding up.” 

Inconsistent responses were also offered to questions about increases or decreases in 
school enrolment due to SIP. At three schools the staff reported that the enrolment at the 
higher grade levels had actually declined because of poaching. “Parents feel that their 
children have a better opportunity to get into a private school because of what they learned in 
SIP at the lower grade levels, but the teaching at the upper grades is not seen as high quality.” 

Contributions to SIP Schools: About half of the SIP parents reported having attended a 
harambee (fund raising activity) since 1994, but SIP parents were no more likely to have attended 
than were control parents. However, 55% of the SIP parents reported providing assistance 
(labour, materials or cash) to the schools in their village this past year and schools in T2 and T3 
were si’gnficantly more likely than control schools to have provided assistance. When asked to 
specify the amount of labour, materials or cash that they had provided, the average amount for T1 
was KSH 394/, for T2 was KSH 715/, for T3 was KSH 895/, and for control schools was only 
KSH 167/. T3 parents provided significantly more than parents in Tl, and parents in T2 and T3 
schools provided signficantly more than parents in the control schools. T3 s contributions may 
have been higher because schools coming into SIP are expected to provide for secure windows 
and doors, and other physical facilities in order to “ qualify” for SIP participation. 

Views of Their Child: Seventy percent of SIP parents reported that their child 

seemed more confident since the school became involved in SIP, and 68% reported that their 
child is more likely to speak freely since SIP involvement. 

Involvement in Their Child’s School: SIP parents were more likely to know what 
the School Committee does than were control parents, and Tl and T2 parents were 
significantly more likely to have spoken with their child’s teacher about their child s 
performance than were T3 or control parents. Seven of the 11 SIP head teachers indicated that 
the number of school visits by parents had increased since SIP began and only one SIP school 
reported a decline in the number of visits by parents. None of the control schools reported any 
change in the number of parent visits. 
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Summary 



SIP did meet its objectives with regard to parents and community. They offered 
training to School Committees in SIP schools and encouraged Committees and individual 
parents to contribute more to the schools -- significantly more than in control schools. 
Seventy percent of the parents reported that their children seemed more confident and were 
more likely to speak freely since SIP involvement. Significantly more SIP head teachers 
reported an increase in the number of school visits by parents since SIP participation 
compared with head teachers in control schools. 

Objective 4: Improve the Cost-effectiveness of the Programme 

In the following discussion, two cost analyses are provided: the first compares the costs- 
per-teacher trained between Phase 1 and Phase 2; the second estimates the costs expended by 
SIP to increase pupils’ test scores over those of the control pupils. 

Comparisons between Phase 1 and Phase 2: The evaluation of Phase 1 asks whether 
the cost-per-teacher-trained could be reduced further, and the Phase 2 proposal states that an 
objective is to seek to improve the cost effectiveness of the programme. In the cost analysis 
for Phase 1, Black estimated the costs in British Sterling. However, since the exchange rate 
has varied markedly from one year to the next, all calculations provided here are in Kenyan 
shillings. During Phase 1, 36 teachers were trained at a total cost of KSH 10,353,000/, 
resulting in a cost-per-teacher-trained of KSH 287,583/. During Phase 2, approximately 168 
teachers, head teachers and other education staff were trained (see below) at a total cost of 
KSH 31,661,000/, an average cost-per-education staff-trained of KSH 188,458/. This is even 
less when adjusted for inflation. Between 1991 and 1994, the GDP deflator increased from 
143.6 to 241.3, reducing the cost per-teacher-trained in real terms by 68% to KSH 112,177/, 
or KSH 175,406/ less than the cost-per-teacher-trained in Phase 1 - a 61% reduction. 

Table 16 

Cost per Teacher Trained - Phases 1 and 2 

Total Costs (Ksh) 

Phase 1 

10.353.000 
Phase 2 

31.661.000 



Number of Teachers Trained 


Cost per Teacher Trained 


36 


287,583 


168 


188,458. 
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During Phase 2 SIP worked with lower primary teachers in 27 schools (3 x 27 = 81) and 
with upper primary teachers (standards 4 - 6) in 13 of those schools (3 x 13 = 39), bringing the 
total number of teachers with which SIP worked directly to 120. During Phase 2 they also 
worked with head teachers, TAC tutors, inspectors, and School Co-ordinators — each of which 
worked directly with teachers to expand and sustain SIP’s effectiveness. We can assume that 
in all, these would amount to roughly 30 additional staff, bringing the total to 150. It should 
be noted however, that Phase 1 teachers continued attending significant numbers of SIP 
workshops during Phase 2 and so continued to benefit from the funds expended during the 
second phase. In addition, the regression analyses show that attendance at these workshops 
was a significant factor in increasing students’ test scores. So, for these purposes, half the 
number of teachers treated during Phase 1 are added to the Phase 2 estimate (+18), since they 
attended workshops but did not receive the classroom-based service provided by SIP. This 
brings the approximate number of professional staff trained during Phase 2 to 168. 

Cost per Increase in Student Learning: In this section, the cost of the amount of 
learning gains produced by SIP are estimated. To do this, the coefficients produced through 
the regression analyses are used, which adjusted test scores for family income, mother’s 
education, and the number of years a teacher had been teaching — all significant influences on 
student learning in most regressions. The coefficients show what pupils’ test scores would be 
in relation to the control group’s, holding these non-project variables constant, and are more 
accurate reflections of SIP’s influence on test scores than if non-adjusted scores were used. 

Total project costs for the six years were KSH 42,014,000/. An estimate of the total 
number of pupils served over the six years is 10,920 (using Black’s figure of 70 pupils per 
class; the average class size for the classrooms in our study is lower, but may not be 
representative of all schools in the Municipality), resulting in a per-pupil cost of KSH 3,847/. 
During Phase 1, approximately 2,520 pupils were served. However, the teachers of these 
pupils continued attending workshops and pupils continued benefiting from the classroom 
materials, so their proportion of the project costs should be greater than would the T2 or T3 
teachers who began receiving SIP services more recently. To adjust for differences in the 
amount of exposure to SIP, costs in T1 are multiplied by 1.5, in T2 by 1, and in T3 by .5. The 
analysis estimates the cost-per-percent gain in test scores for each of these groups at the 
standard 3 level, but at the standard 6 level, combines the scores and costs since all teachers at 
that level began participating in SIP during Phase 2. Test score gain is defined as the 
difference between control and treatment school mean scores across all tests for that level 
(with statistical adjustments for family income, mother’s education, and the number of years a 
teacher has been teaching). The total percentage points gained was divided into the total 
percentage points possible across all tests. So, for example, the T1 pupils scores were an 
average of 65 percentage points higher than the control pupils’ scores. However, there were a 
total of 400 percentage points possible, so 65 is 16.25% of the total percentage points 
possible. 
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Table 17 shows that the cost-per-percent-increase for a T1 pupil KSH 361, or $6.56. T2 
pupils only had an average of a five percent increase over the control pupils, resulting in a 
cost-per-percent-increase per pupil of Ksh 769, or $13.98. Adjusted scores for T3 pupils were 
the same as those for T2, but since their exposure to the SIP treatment was for a much shorter 
length of time, the cost-per-pupil is adjusted by half, resulting in a cost-per-percent-increase 
per pupil of only KSH 385/, or $6.99. Since the adjusted gains for pupils in T1 were so much 
higher, their per-pupil cost is lowest of the three groups, but close to that of the T3 pupils who 
have been in the programme for only a short period of time. 



Table 17 

Cost Per Percent Increase in Test Scores* 



Group 


Percent Increase 


Cost Per Pupil 


Cost Per % Increase 


Cost in US$ 


Per Pupil 


KSH 


Per Pupil - KSH 


Per % Increase 


Standard 3 
Tl 


16% 


5,771 


361 


$ 6.56 


T2 


5% 


3,847 


769 


$13.98 


T3 


5% 


1,923 


385 


$ 6.99 


Standard 6 
(T1+T2+T3) 


13% 


3,847 


296 


$ 5.38 



* Test scores across all tests given at that class level 

Standard 6 pupils achieved a 13% increase across all tests, resulting in a low per-pupil 
per-percent cost of KSH 296/, or $5.38 - lower than that of any of the standard 3 groups. The 
average 1993 (midpoint of the project) per-pupil expenditure at the primary-level in Kenya 
was KSH 2,131/. So, SIP spent about 15% more for each additional percent increase in test 
scores for pupils in Tl, T3, and the standard 6 pupils, but spent 36% more for pupils in T2 
schools. 



The cost of expanding the SIP programme to other municipalities or districts is not 
calculated since our recommendations do not include maintaining SIP, as it is currently 
conceived, as the preferred treatment. 

In most cost-effectiveness analyses, one or more approaches are generally compared. 
However, the only comparison groups available for this study are either the control group, 
which represents “business as usual,” - what happens when no external intervention is 
provided, or comparisons of costs and accomplishments between Phase 1 and Phase 2. It 
would be instructive if AKF were to systematically vary the approaches tried within a project 
when they begin a new initiative so that a comparison could be made of various approaches to 
improving schooling. An example of such a comparison is briefly reviewed here for 

illustrative purposes. 
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A 1991 study compared the cost-effectiveness of three approaches to teacher training in 
Sri Lanka and > found that distance education was substantially more cost-effective than 
residential training programmes offered in colleges of education or teacher training colleges - 
about 4.5 to 6 times more cost-effective. ,v Although the distance education programme was 
not the most effective, its low comparative costs made it a viable policy option, since teachers 
in the distance programme continued teaching, while the other two groups did not. However, 
the study found that teachers' knowledge of what they had learned in the training substantially 
declined over time. 

Print-based distance education often has been shown to be cost-effective, but usually 
suffers from exceptionally high attrition rates since it requires more discipline on the part of 
the learner. An alternative distance approach using multimedia-based training is discussed in 
the recommendations section of this report. Multimedia (i.e. CD-Rom) offers some of the cost 
benefits of print-based distance but is more engaging to users and thus can preclude high 
attrition rates while maintaining consistently high quality not controlled in training-of- 
trainers-type approaches as used in SIP. 

Summary 

SIP significantly reduced the cost-per-teacher (educator) -trained by 61% during Phase 
2. However, SIP’s ultimate benefits in terms of pupil learning were less dramatic and fairly 
costly. The cost of increasing a pupil’s test score by one percent ranged from about KSH 300/ 
to 800/, or $US 5.00 to $14.00 per pupil. Relative to the standard Kenyan 1993 per-pupil- 
expenditure of KSH 2131/, SIP spent about 15% more for each additional percent increase in 
test scores for pupils in groups Tl, T3 and the standard 6 pupils, but 36% more for pupils in 
T2 schools. These figures are fairly high and probably not viable costs for a government 
agency with limited resources to incur, but they can serve as a benchmark for comparing with 
other approaches SIP may try to increase learning. 

Describe the Project Implementation Strategy and the Extent to Which it has Been 
Successfully Effected. 

The project’s implementation strategy was described in detail in the first section of 
this report and generally coincides with that described in the project proposal. However, one 
of the features of the strategy that changed from the proposal description is that the 24 Master 
Teachers were no longer used. Another is that a monthly newsletter proposed was not 
produced. More significantly however, is that the proposal specifically intended to avoid the 
“dilution of the original ‘message’ by” not using the cascade model of teacher training. 
Based on discussions with the SIP POs about their own training to be trainers and to provide 
classroom-based coaching, it appears that some of this dilution may have still occurred. The 
issue of the training of SIP staff addresses another of the study questions, to identify the main 
constraints in the implementation of Phase 2. 

Evaluation of Phase 2 of the Kisumu School Improvement Project Capper, Nderitu, Ogula 




91 



SIP Staff Training: SIP Programme Officers (POs) were selected from the ranks of 
classroom teachers within the Municipality and most had been considered exemplary teachers. 
They were the front-line contact with teachers and head teachers in SIP schools and are the 
individuals who worked most closely with teachers, providing demonstration lessons, 
explaining the rationale for various approaches, commenting on the teacher’s lesson, 
negotiating new teaching strategies, and working with the head teacher and School Co- 
ordinator to ensure that follow-up support was provided and school practices were supportive 
of the SIP agenda to promote more child-centred teaching and learning. The evaluation team 
met several times with the Programme Officers to obtain their views regarding several 
dimensions of the project and the training they received that would allow them to provide 
teachers with high-quality training and technical assistance. The following information is 
based on these discussions. 

The POs reported that they received no training for their jobs during the first phase of 
the project. However, at the beginning of Phase 2, when the new Project Director arrived, 
they received training in leadership styles, collaborative decision-making, and interpersonal 
skills (e.g., how to begin coaching sessions by telling teachers about the positive things they 
see them doing), among other topics. 

In the beginning of Phase 2, the POs attended a one-week intensive workshop 
conducted by a professor from St. Mark and St. John’s College (affiliated to Exeter 
University) in the U.K. in classroom arrangement and management, discipline, and tracking of 
pupils’ progress. In December 1993, they received training in child-centred methods, learning 
centres, reading, and imagined literacy (i.e., how to promote children’s development of 
literacy skills from their environment and how to make classrooms literature rich.) The 
Project Director conducted much of the training for the POs, including training in the whole 
language approach to literacy. The POs reported that most of the training was conducted in a 
lecture format with numerous activities and group discussions. Most of what they learned was 
somewhat familiar to them from their pre-service coursework, although they commented that 
the SIP training allowed them to see how the topic was carried out in other contexts. 

When asked, they reported that they were not offered opportunities to read or discuss 
original source material in conjunction with any of their training sessions, such as research 
articles or summaries of research on how children learn to read. Instead, they were given 
handouts that summarized the points covered in the workshop sessions. Moreover, they 
reported that they were not provided with opportunities to reflect on what they learned as a 

group. 
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When asked to rate the training they received to do their jobs, the evaluators created a 
rating scale with endpoints of 1 and 10. Point 1 represented training where only the theory 
was explained and point 10 represented opportunities to master the content, skills and 
procedures in which they were expected to train and coach teachers. A point of 10 would 
have included in-depth coaching of the POs until they had mastered the essential skills and 
concepts. The mid-point of 5 represented training that provided theory, opportunities to 
discuss, and group activities, but no observation, coaching or feedback as they worked with 
teachers. All POs present at the discussion with the evaluators rated their training between 4 
and 6. They indicated that they would have preferred training and coaching that gave them 
time to reflect in groups, to read and discuss original source documents, and to have the 
expert(s) observe them in the classrooms as they worked with teachers. 

They did report that they had weekly meetings in which each PO had prepared a 
written report of what had happened in the field during that week. They would discuss their 
reports and the Project Director would offer suggestions for how to deal with various 
situations. They also reported that they received feedback and support from the Project 
Director on the workshops they conducted for teachers, and that the Director would review the 
workshop materials with them prior to the workshops. In addition, most of the workshops 
were videotaped and the videotapes were reviewed and critiqued by SIP staff. 

They reported that they had not received any guidelines or training in what to do when 
they visited a school. Nor were they taught how to do a demonstration lesson or give 
feedback to teachers. 

In late 1994, AKF sponsored eight of the POs, an inspector, a TAC tutor, and 12 
teachers from the AKES network in East Africa to participate in a postgraduate degree 
programme with Exeter University in Great Britain. The coursework consisted of five 3-week 
sessions and a research project. The first session was on learning theories, including the 
Social Constructivist theory of Vygotsky; the second was on teacher education; and the third 
was on curriculum, comparing Kenya’s curriculum with those of Great Britain, the Caribbean, 
Fiji, and several other African countries. Each PO then selected a research topic which they 
were still conducting or writing during the evaluation visit. The topics selected were: 1) the 
language of instruction, 2) the head teacher’s contribution to the quality of teaching, 3) the in- 
service needs of primary school teachers and the capacity of the TACs to meet those needs, 
and 4) special needs students. All of the POs enrolled in the Exeter programme gave it the 
highest possible rating and were very enthusiastic about it. 
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However, even this programme was not tailored to the day-to-day responsibilities of 
the POs. A mofe appropriate degree programme might have been designed that would have 
developed their expertise in ideas and skills such as: clinical supervision; higher-order 
thinking and questioning strategies; a whole language approach to teaching reading, writing, 
and language development, especially aimed at youngsters learning in a second or third 
language; applying child-centred, constructivist teaching strategies to each of the core subject 
areas; classroom-based coaching of teachers; etc. 



Observations of SIP Programme Officers Working in Classrooms and 
Conducting Training: The evaluators asked to observe the SIP POs working with teachers 
in their classrooms since this was such a critical aspect of the SIP treatment. In addition, 
several videos of SIP workshops were previewed and one live training session was observed. 
The observations of the POs confirmed the inadequacy of training they received to conduct the 
classroom-based coaching sessions and revealed deficiencies in the quality of the project’s 
implementation. 

Observation - PO # 1 : The Programme Officer asked the standard 3 teacher if we 
could observe the teacher teaching a lesson. This was agreed and we sat in the back of the 
room and observed for about 35 minutes. In the lesson, the teacher did not involve children in 
the teaching and learning process nor did she have them work in small groups. She did not 
have the pupils use textbooks and did not interact with the children as she moved around the 

room. 

tjjg lesson was over, the PO discussed the lesson with the teacher and offered 
several suggestions. A summary of the PO s comments follows. 

That was a nice lesson, but there were some children who were not following. It 
might be better if children had an opportunity to form their own words. If children finish their 
work early, you could give them cards to work with and could have them write words on the 
chalkboard’. Also, the children were seated in homogeneous groups , but it would be better to 
put them in heterogeneous groups. You’ll need more desks for that. 1 11 talk to the head 
teacher about it. The PO then told the teacher that he would plan a lesson with her on Monday 
for her to teach on Tuesday. 



Evaluation of Phase 2 of the Kisumu School Improvement Project 



Capper ; Nderitu, Ogula 



Observation - PO #2: The PO explained to the teacher that she would like to do a 
demonstration' lesson: The teacher agreed and seemed eager to see it. The PO started the 
lesson by asking, “Tell me any word in English that begins with the letter B.” Before the 
children could respond to the question, she asked them the same question in Luo (the local 
language). Although the PO was enthusiastic and appeared to know the subject matter well, 
almost all of the questions asked were closed questions requiring only simple one-word or 
short-phrase responses, with little or no complex thinking. Children were seated in groups but 
there was no group interaction and very few children seemed to be engaged with the lesson. 
No teaching aids were used and choral answers were encouraged. After the lesson the PO 
asked the teacher for comments about the lesson. 

At a subsequent meeting with the whole group of SIP POs, the evaluators asked if 
they had received any specific training for working with teachers in the classroom. They 
indicated that they had not received training in the procedures of conducting demonstration 
lessons, evaluating teachers’ lessons, or giving feedback to teachers. In addition, there does 
not appear to be any written guidelines for use during school-based TA. 



Observations of SIP workshops: Fortunately, SIP has made videotaped recordings of 
almost all of the workshops they have conducted and these were previewed by one member of 
the evaluation team. However, other than the Project Director, only one of the current POs 
was depicted conducting any SIP training and that session was in the Luo language. The 
Project Director was leading many of the workshop sessions, and in almost all cases, the 
participants were seated in groups and during various stages of the workshops, were involved 
in activities or group discussions. Most of the other presenters were either former POs who 
were no longer with the project or external experts. 

One hour of a live two-hour reading workshop session was observed, attended by 
seven teachers and led by two of the POs. There was a considerable amount of time spent 
addressing the importance of getting to know the pupils, and trying to connect the lessons to 
what the pupils already know, experience or feel, and another portion of the workshop devoted 
to the use of learning aids. Although there was some question-and-answer dialogue with the 
participants, the POs did most of the speaking and, during the one hour observation, there 
were no group activities or discussions. Strangely, they spent only about two minutes on 
methods for teaching reading, citing only the look-and-say and phonics approaches and saying 
nothing about the whole language approach. However, this session may not have been 
representative of those that are offered for a longer period of time. The POs seemed pressed 
to cover the material in the short time available. 
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Staff Turnover: SIP had a very high rate of staff turnover. We did not realize this 
until we began viewing the videotapes of the training sessions and did not recognize any of 
the trainers. We were told that many of these trainers were previous POs who had moved on 
to other positions. Although it is not clear why the turnover was so high, each time a staff 
person leaves, a new person must be trained and the effectiveness of the programme is 
diminished. This is not a cost-effective approach to project management. AKF might wish to 
review their projects to assess the extent of staff turnover, and if it is determined to be 
unreasonably high, to try to identify the reasons and take steps to reduce staff turnover. 

Summary 

There were several main constraints to increasing the project’s success in changing 
teachers’ behaviours, in improving schools’ effectiveness, and in increasing student learning. 
Some of these were due to inadequate project management and some were beyond the 
project’s control but within their realm of influence: 1) ) SIP waited until very late to work 
with head teachers; 2) SIP did not focus on the whole school as the unit of improvement; 3) 
SIP staff were not adequately trained; and 4) there was a high rate of staff turnover. 



Other constraints included: a high rate of school and MEO staff transfers; a high rate 
of school closings for multiple examinations, athletic events and music festivals; and a high 
rate of teacher absences, in part due to the high rate of AIDS-related illnesses and deaths in 
Kisumu but also due to lax supervision on the part of the head teachers and the MEO. 
Although SIP staff did encourage school and Municipal staff to address these issues, there was 
little evidence that they were successful in evoking significant improvements. 

Assess the Extent to Which the Project Addressed Itself to Matters Concerning 
Policy and to What Extent Key Players in Education Contributed to the Improvement 
of Education Standards 



In many ways SIP worked closely with the MEO and even the Municipal Council to effect 
change in Kisumu’s schools. They began by having the project s governance committee 
chaired by the Municipal Education Officer and having zonal inspectors, TAC tutors, a parent, 
and a teacher training college lecturer on the project’s eduactional guidance committee They 
worked quite closely with the MEO’s office throughout Phase 2 and influenced replacement 
and upgrading of selected TAC tutors and inspectors, as well as revision of the school 
inspection form used by the inspectors. 
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They also established a system-wide process of collection of funds from parents for 
centralized bulk purchasing of instructional materials, encouraged the establishment of TACs 
in each of the Municipality’s zones, and contributed to the equipping of these TACs. The 
District Primary Schools Inspector instituted panel inspections of schools designed to ensure 
that schools put into practise the new ideas emanating from SIP workshops. The Municipal 
Council has begun to invite the SIP Project Director to their meetings, demonstrating the 
value that the Council attaches to the project, and this same Council voted to transfer the SIP 
Office to the MEO’s complex of offices and provided funds for the remodeling of office 
space. 

However, it appears that SIP had little impact on several important elements of 
Municipal policy, such as the high rate of staff transfers, frequent school closings, and high 
rate of teacher or head teacher absences. 



Were SIP’s Materials Useful and of High Quality? 

A prominent feature of SIP’s focus was on instructional materials for use by teachers 
in the classrooms. It appeared that the Project Director favoured these materials above other 
elements of the SEP programme, such as teacher training and coaching. The materials 
generally were made of construction paper and were created into charts and maps and either 
hung on the classroom walls, set up as learning centres around the classroom, or used as 
learning aids by teachers and pupils. For example, SEP taught teachers to make small cards 
with words on them that the children could use to form sentences in small groups. Or in math, 
they were taught to use sticks and bottle caps to help children learn the concepts underlying 
the basic mathematical operations. 

The materials had an invigorating and warming effect on the classrooms, in stark 
contrast to non-SEP classrooms, often within the same school. Everyone loved the materials 
and were constantly calling for the provision of similar materials for other teachers in the 
school. And, in fact, SEP worked with the MEO to establish a system for doing just that. 
Each parent was to pay KSH 50/ per child per year to be collected by the head teacher and 
deposited into a central account managed by the Municipal Council who would then purchase 
materials in bulk at discount rates. The intent was to make the collection and use of the funds 
transparent. However, a copy of a memo from the new Project Director to AKF Nairobi 
headquarters dated October 1996 reported that in some instances KSH 100/ was collected 
instead of the prescribed Ksh 50/ and expressed concern that “A lot of it [these funds] as 
been collected but not put to right use and has been documented by KSIP.” It appears that 
there are problems that make this process less transparent and effective than was originally 

intended. 
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The regression equations discussed earlier suggest that either the materials and/or the 
coaching provided by SIP (as represented by the treatment group variable) contributed to an 
increase in test scores — at least at standard 3. It was not possible, however, to sort out the 
relative contribution of materials versus coaching. 

Although we often observed the SIP teachers using the learning aids (sticks, bottle 
caps, etc.) in their lessons, we seldom saw the children using them in small groups. We never 
observed them using the learning centres, nor did we see teachers refer to the charts on the 
walls. However, it is quite likely that the teachers may have felt pressured by our visits and 
reverted to teaching strategies with which they felt most comfortable. Another observation 
was that all of the materials looked the same. It did not appear that most teachers had used the 
concept of creating materials to fashion some of their own, or to have children create their 
own. For example, according to the whole language approach of teaching reading and 
writing, teachers might have the children work together to create their own stories, which the 
teacher could then form into a large-print book and use as reading material. It might be useful 
for SIP to place a greater emphasis on showing teachers how to be more creative in producing 
their own materials. 

At each of the SIP workshops, handouts were provided to participants. Eventually, 
these handouts were collated and bound together into a newsprint book — without a cover. 
Although the evaluators found the quality of these materials less than coherent and were 
inconsistent in their style, clarity, and utility, most of the SIP teachers reported that they 
thought the handouts were clear and helpful and that they used them to refresh their memories 
of what they learned in the workshops. However, few of the teachers reported seeing the 
bound book and none pulled out their materials to show us. They did indicate that they would 
have preferred a more step-by-step description with examples of some of the things they 
learned. Only two SIP teachers felt that the ideas in the workshops and materials were 
unrealistic for the teaching situation in Kisumu. 



Since AKF works in similar projects across many sites, it may be worthwhile to 
centrally produce manuals or handbooks for the various audiences that these projects address, 
such as teachers, head teachers, inspectors, parents, etc. In this way, it would save the cost of 
having each project repeatedly devote time to creating such materials and could ensure high- 
quality, meticulously clear products based on the latest research in teaching and learning. 
This would not preclude projects from developing additional materials to suit local needs. 
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An example of one of the SIP-produced materials that could be improved is the 
observation checklist for head teachers’ used in observing teachers. As mentioned earlier, the 
form is far too long and complicated. Both teachers and head teachers can feel overwhelmed 
if they feel they are aiming for too many targets. It is better to focus on changing a few highly 
important teaching behaviours (such as having students work and discuss in small groups, or 
asking questions that encourage children to think) and ensuring that the teachers master these 
behaviours, rather than introduce too many behaviours that make them feel it is hopeless to 
even try. 

Exams: SIP also worked with teachers and other educators to produce practice exams 
leading to the national leaving examinations. As discussed more fully in another section, 
these exams, in the evaluators’ views, are not at all child centred and actually undermine the 
child-centred teaching strategies SIP is trying to promote. Moreover, there are very few items 
on the exams that measure higher-order thinking skills and there are also serious problems 
with the exams, despite the fact that they were reviewed by the Kenya National Examinations 
Council. It is quite likely the items were reviewed solely for their statistical properties and not 
for their pedagogical validity. For example, the standard 1 science questions were in English, 
despite the fact that the children have not yet learned English! One of the items even has the 
word germinate — far too difficult a word even for native English speakers at that age. A 
standard 4 reading comprehension passage seemed entirely inappropriate for young children. 
It was a short story about a “ very black man” who had fierce eyes and chased children asking 
for food until the men in the village chased after him with spears and told him not to come to 
the village again. In the last sentence, the women who went for water could not find their 
children, suggesting that the hungry man had kidnapped them. Not only does this passage not 
show compassion for the poor and hungry, and instils fear in the children, it also suggests that 
being very black inclines one to be dangerous and someone to be afraid of. Another item has 
four words that are to be arranged to form a sentence, but actually has two correct answers. 
Only one is allowed. These examinations should be revised to be technically and 
pedagogically valid and supportive of child-centred teaching strategies before they are used 
again. 



Research and Monitoring: The Phase 1 evaluation included a recommendation to 
increase attention to systematic self evaluation. SIP was exemplary in this regard. It hired an 
experienced and skilled researcher and conducted several studies throughout Phase 2. In most 
cases, the results of these studies were used to help shape next steps. As mentioned earlier 
though, it would have been useful to monitor the extent to which teachers were integrating and 
retaining child-centred teaching strategies into their instructional repertoire. 
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Summary 



SIP significantly enhanced the classrooms in which it worked with instructionally- 
useful materials, and wisely established a mechanism to ensure that all teachers within the 
Municipality would have access to similar materials. However, materials used to convey 
teaching skills and concepts were inconsistent in their quality and dissemination and should be 
centrally produced by AKF to ensure consistency in quality and save time and funds across 
AKF’s many school-improvement projects. The examinations developed by SIP Kisumu are 
not supportive of child-centred teaching and should be discontinued until revised. SIP’s 
research and monitoring efforts were exemplary and useful to the project. It may be helpful 
for SIP or AKF Nairobi to work with the MEO to encourage continuation of such research and 
monitoring activities by MEO staff. 

VIEWS OF SIP 

Information was gathered from teachers and head teachers regarding their views of 
SIP overall and the technical assistance and training provided by SIP. Pupils in both treatment 
and control schools were also interviewed to determine if SIP had had an impact on their 
views of schooling, their teachers or learning. Where appropriate, comparisons are made with 
control teachers, head teachers, and pupils. 

Teachers’ and Head Teachers’ Views of SIP 

Statements and perceptions below were gathered during interviews conducted with 
each of the teachers observed and the head teacher at that school, as well as from the focus 
group sessions with the school staff. A range of questions was asked about their perceptions 
regarding the influence of SIP. Control teachers and head teachers were asked similar 
questions where possible, but without reference to SIP. For example, Have you (the head 
teacher) changed the way in which you work with teachers over the past 3 years?” 

In what ways have the SIP Program Officers been helpful to you in your teaching/role as 
head teacher? 

(Head Teachers) 

■ Offered classroom-based training for teachers. 

■ Encouraged the lower primary teachers to put more effort into their teaching so that the 
head teacher could pay more attention to the upper primary. However, some of the 
other teachers got resentful when they didn’t get the materials. 

■ Provided manila papers at affordable prices. 

■ Advised us how to manage the staff and evaluate teachers. 

■ Have been very co-operative. Anytime we need help, they always come through. 

■ Run courses; demonstrate lessons in class; make and give us resource materials. 
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(Teachers) 

■ The in-service and videos were helpful. 

■ Reminded me about child-centred teaching methods. We were taught this in teacher 
training college but most of us have forgotten what we learned. 

■ Demonstrate lessons and help us to make teaching aids. 

■ Guide us in how to implement the curriculum. 

■ Available for consultation and provide resources. 

■ Have guided me on how to approach certain topics such as practicals, games and 
calculators in math. 

We asked the POs if they had seen any change in attitudes in schools since they began 
working with SIP? They responded, “The attitudes in the SIP schools are much more positive 
than before SIP began. Teachers are more positive and open and much more friendly to the 
children. Parents are more aware because their children are more likely to talk about what 
happens at school. In addition, School Committees and PTAs are now more involved in their 
schools. When SIP began, the teachers rejected the SIP staff, but now they come to the team 
members for help. Teacher attendance at workshops has increased. In the beginning, only 
one or two teachers would attend, and now most workshops are full.” 

Has SIP made a difference in the quality of teaching and learning in your school? 



One school reported that teachers had worked together across grades to prepare 
common schemes of work, and staff in several schools reported that the children are now more 
comfortable expressing themselves. Some of the strengths of SIP that were offered included: 

■ They’ve improved the standards. 

■ Lots of encouragement. 

■ Practical follow-up with action. 

■ Promote grouping of children and better questioning. 

■ Grouping allows the weaker children to leam from the faster ones and allows teachers 
time to work with the weaker pupils. 

■ Pupils can now read better. Having teaching aids and library books helps. 

■ We used to just lecture to the children and give them a little work. Since SIP, there s 
more work and involvement for the children. 

■ The learning centres help the teacher to have an easier time. 
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Pupils’ Views 



A random sample of 15 standard 3 and all of the standard 6 pupils from each of the 
study classrooms completed a survey asking about attitudes towards school and learning, 
whether they read at home, whether they understand what they leam, etc. Pupils in standard 3 
were interviewed in their first language by trained data collectors while pupils in standard 6 
completed the survey after the evaluators completed the classroom observations. One 
observer fluent in the local language, stayed with the pupils to give them instructions and 
answer questions. Results are reported below. 

Do you read at home? In standard 3, 77% of the pupils reported that they read at 
home, but only 32% were able to name a book they had read (textbooks were not counted). 
Almost all of the standard 6 pupils reported reading at home and about 75% of the pupils in 
T1 and T2 could name a book they had read, while only 35% of the pupils in T3 could name a 
book, but 84% of the pupils in the control group were able to do this. 

Do you like school? Several questions were asked to assess pupils’ attitudes toward 
school There was little variability in pupils’ responses. Almost all pupils at both class levels 
reported liking school, liking to leam, having a teacher who likes them, and liking their 
teacher, although only about three-fourths of T1 pupils reported liking their teacher compared 
with 95-98% in the other groups. However, when asked whether they would rather stay home 
than come to school, about 25% of all of standard 3 and the standard 6-T3 pupils responded 
“Yes,” while only 12% of T1 and T2 standard 6 pupils responded “Yes.” Only 8.5% of the 
Siaya pupils said that they would rather stay home. 
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Locus of control: Research indicates that one is more likely to perform well at 

something if one feels in control over that area of one’s life, and of one’s life overall.”’ For 
example, individuals who feel that luck or fate works against them are not as likely to take the 
steps needed to ensure that things work as they hope and intend. Several questions in both the 
pupils’ and teachers’ surveys were included to assess locus of control, for example, “When I 
pass a test, it’s because 1) I worked hard, or 2) I have good luck (bahati/hawi). 

Almost all of the pupils in all groups responded that they will get good grades if they 
work hard and that if they fail an examination it is because they did not study hard enough -- 
not because the teacher didn’t like them. However, at the standard 3 level, 14% of the T3 
pupils answered that they passed a test because they were lucky instead of because they 
worked hard. The figures were all below 10% for the other groups. 

Do you understand what you learn? Almost all the pupils indicated diat they 
understand what they are learning - or understand some but not all. Almost none indicated 
that, “I just memorize it - 1 don’t understand it.” 

Do your parents pay your teacher for tutoring? About 50% of the standard 3 
pupils sampled in groups T1 and T2 reported that their parents paid their teacher for extra 
study, compared with only 20% for T3, and 26% for the Siaya control schools. There was a 
great deal of variation in the responses to this question, ranging from 0 to 88% per school. 
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At the standard 6 level, the differences are even more marked. Three-quarters of the 
pupils in groups T1 and T3 reported that their parents paid their teacher for extra study while 
50% of those in group T2 reported paying and 29% in the control group pay The regression 
analyses indicated that tutoring did not have a significant influence on students tes scores 
and so may not be a worthwhile expenditure for parents, although it is possible that scores 
may have been lower without the tutoring. Pay for tutoring is against national and municipal 

policy, but clearly is not enforced. 
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Number and Rating of Workshops Attended 

Number attended: Each teacher and head teacher was asked to indicate the number of 
SIP workshops they attended, as well as workshops sponsored by the TACs and by SPRED. 
SPRED is another teacher professional development initiative sponsored by the ODA and 
focuses on training in the subject content. 

Table 15 shows that in groups T1 and T2, standard 3 teachers attended an average of 
seven more workshops per teacher than did standard 6 teachers. However, in the T3 schools, 
which just began participating in SIP in January 1996, both standard 3 and 6 teachers attended 
12 workshops, about half as many as those attended by T1 standard 3 teachers (25) and three- 
quarters of those attended by T2 standard 3 teachers (16). All SIP teachers at the S3 level 
attended more workshops than did the control teachers (9), but at the standard 6 level, control 
teachers attended more workshops (12) than did the T2 teachers (9) and as many as the T3 
teachers (12). Most of the workshops teachers reported attending were SIP-sponsored 
workshops. The influence of the number of workshops attended on test scores was discussed 
in the regression analyses reported earlier and found to be a consistently significant and 
positive factor in student achievement. 



Table 15 

Mean Number of SIP and Non-SIP Workshops Attended 
by Teachers by Group and Class 



Group (N) 


S3 Teachers 
SIP Non-SIP 


S6 Teachers 
SIP Non-l 


T1 (4 teachers per level) 


23 


2 


14 


4 


T2 (4 teachers per level) 


16 


0 


7 


2 


T3 (3 teachers per level) 


11 


1 


11 


1 


C (4 teachers per level) 
* One school missing. 


NA 


9 


NA 


12 



Ratings: Teachers and head teachers in all treatment groups rated the workshops as 
being helpful and influencing their practice. Most consistently rated the workshops between 
helpful and very helpful (4.5 on a 5-point scale). 
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Summary of Findings 

In this section, we briefly review the findings of this study and then consider the 
likelihood that SIP’s efforts will continue once external funds are no longer available and the 
management of SIP is absorbed by the Kisumu Municipality. 

SIP worked with lower primary teachers in 27 schools and upper primary teachers in 
13 schools; they provided instructional materials to over 100 classrooms and teachers and 
trained teachers in their use; quite late in the project they began working with head teachers; 
they met with the parents and School Committee for each school to introduce them to the SIP 
model, to enjoin their participation and support and to encourage contributions toward the cost 
of materials; they established a close and productive relationship with the MEO and the 
Municipal Council resulting, in part, in replacement and training of many of the inspectors 
and heads of Teacher Assistance Centres; they instilled an ethos of school improvement and 
attention to the concepts of child-centred teaching throughout the Municipality. 

Although SIP teachers exhibited substantially more child-centred teaching behaviours 
than did the control teachers, in most cases, their use of these behaviours was limited and 
often shallow. The more important, yet difficult, behaviours such as asking questions that 
engage pupils in thinking analytically, having pupils be active learners working in small 
groups, encouraging children to express themselves and explore ideas or use language to 
communicate and understand, were rare in most classrooms observed. However, a few 
teachers did exhibit exemplary teaching behaviours. Although SIP teachers did engage in a 
more child-centred approach to teaching, these behaviours did not appear to have a positive 
influence on test scores at either grade level. 

At the standard 3 level, when non-treatment factors were controlled for, being in a SIP T2 
school has a positive effect on test scores in English and math, as does the number of SIP 
workshops a teacher attended. Teachers’ classroom management behaviours were also a 
significant influence on English scores. The most consistently influential variable was 
number of years a teacher has been teaching. At the standard 6 level, when non-treatment 
variables are controlled for, SIP had no influence on any of the test scores when compared 
with those of the control schools. 

Pupils in both treatment and control groups performed reasonably well on the short- 
answer and multiple-choice tests, suggesting that they are learning what is in the curriculum in 
the way that it is taught in the curriculum and textbooks, although standard 6 mathematics 
scores were weak. Scores were exceptionally poor on all open-ended tests for all groups. 
There is clear evidence that children are not learning how to read, write, or communicate in 
English or Kiswahili. Nor are they learning to apply simple mathematical concepts and skills 

to real-life types of problems. 
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SIP parents and School Committees contributed significantly more financial resources and 
in-kind services. to their children’s school than did parents in control schools. Seventy percent 
of the SIP parents reported that their children seemed more confident and were more likely to 
speak freely since SIP involvement. 

Interviews with teachers and head teachers revealed very positive attitudes toward the SIP 
project and Programme Officers. Teachers seemed to believe in the value of child-centred 
teaching strategies, but were reluctant to adopt fully these strategies because they felt a 
pressure to cover the curriculum and ensure that pupils were prepared to take and succeed in 
the national primary certification examinations. They were not able to reconcile these goals. 
In addition, teachers may have misconceived the meaning of child-centred teaching and 
learning, which further inhibited their use of these strategies. 

SIP significantly enhanced the classrooms in which it worked with instructionally-useful 
materials, and wisely established a mechanism to ensure that all teachers within the 
Municipality would have access to similar materials. However, the examinations developed 
by SIP are not supportive of child-centred teaching and should be discontinued until revised. 
SIP’s research and monitoring efforts were exemplary and useful to the project. 



SIP staff were not trained to a sufficient level of mastery to do their jobs. They reported 
that they had not received any guidelines or training in what to do when they visited a school, 
nor were they taught how to do a demonstration lesson or give feedback to teachers. Other 
shortcomings include: SIP waited until very late to work with head teachers; they did not 
focus on the whole school; they did not appear to bring project teachers to a sufficient level of 
skill mastery in using child-centred techniques; and there was a high rate of staff turnover. 

How Sustainable are These Results Likely to be Once the Program is Turned Over to 
the Kisumu Municipality? 

Given this mix of findings regarding SIP’s effectiveness, what is the likelihood that 
SIP-trained teachers will continue to use and expand their child-centred teaching strategies 
once AKF’s support is withdrawn? Moreover, will the other dimensions of SIP s work 
continue and grow under the new leadership of the Kisumu Municipal Education Office? 
AKF’s support was scheduled to end in July 1996 and the staff and activities of the project 
were to be absorbed by the MEO. Several factors suggest that sustainability may be a 
challenge. In addition to the evaluators’ views on this issue, teachers, head teachers, and 
school committees, were asked to speculate about the likelihood of sustainability and the 
factors that would contribute to, or undermine, the chances of success. Suggestions for 
increasing the likelihood of sustainability are contained in the recommendations at the end of 

the report. 
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Staffing: Only one-out-of-eight SIP Programme Officers, one secretary, an office 
messenger, a stockroom clerk, and three drivers have been transferred to the MEO’s payroll 
and supervision. Another PO became a zonal inspector. The new Project Director is being 
maintained on the AKF payroll until December 1996. There are several administrative 
problems that have prevented absorbing more of the POs into the MEO. If the POs are not 
employed by the Teachers’ Service Commission (TSC), then they cannot be employed by the 
MEO since the TSC pays their salaries and the MEO’s office only has budgetary approval for 
the staff listed above. 

In October 1996, it was reported by the current SIP Project Director that new 
Programme Officers would be hired to bring the total number of POs working for the MEO to 
nine. The individuals hired will be drawn from the ranks of practising teachers who would 
continue drawing their salaries from the TSC. These teachers will need to be trained to 
provide training and technical assistance services. The training of the previous POs was a 
rather lengthy process that continued throughout Phase 2, and there is little evidence that we 
were able to gather to verify the proficiency of the POs in these respective tasks. The 
feedback from teachers and head teachers regarding the POs as a group, indicates that they are 
quite skilled at positive and productive interactions with school staff, but the very limited 
observations of two PO classroom visits and one workshop session by members of the 
evaluation team suggests that they may benefit from additional training and support in 
constructivist training and coaching strategy. 

It is our view that the POs should be masters of the skills which they are exhorting 
teachers and head teachers to adopt and that any project of this sort should design training for 
project staff to ensure mastery of all of the component skills and subskills, including those 
aimed at the ultimate target audience - children (e.g., child-centred teaching methods, 
questioning strategies, whole language approach to reading, clinical supervision, etc.) and 
those skills needed to perform their jobs (e.g., participatory training, classroom coaching, 
etc.). In addition, the recent research literature suggests that staff developers also need 
expertise in systems thinking, group facilitation skills, long-range planning, and results -driven 
education. This would require articulating a detailed curriculum for PO training as well as a 
set of measures for each of the skill areas to use in assessing their mastery. What can happen 
without this level of quality control is that they may have an understanding of a skill or 
concept that is not-quite-right, and as it gets disseminated down through the various levels of 
training and TA, the quality control steadily dissipates. The current approach to SIP training 
and TA is likely to have suffered from quality dissipation. In addition, there was considerable 
turnover of POs throughout the project, causing the staff training process to be virtually 
ongoing. 
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Resources: While the SIP project staff had access to three vehicles that allowed them 
to get to schools and workshops on a daily basis, the current effort has only one vehicle. The 
POs are not likely to be able to accomplish their intended goals of providing training and TA 
to teachers if they are not able to get to the schools regularly. 

An office within the MEO complex has been renovated and furnished to house all of 
the POs, Municipal inspectors, and other Municipal education staff. This represents a 
significant commitment on the part of the Municipal Council and the MEO and may promote 
the spread of SEP views and strategies through collegial interaction with other MEO staff. 

SEP has and continues to offer training to the TAC tutors and inspectors. While this 
may enhance the consistency of the SIP message, the inspectors are not likely to offer much in 
the way of regular assistance to schools since most of the teachers and head teachers reported 
that the inspector came to the school only once-a-year. At the least, the inspections will not 
work against what SEP is trying to accomplish as was the case at earlier stages of the project, 
but it is not likely that they will be a major contributor to sustainability. However, since the 
TAC tutors offer training and TA to schools, they are more likely to be able to make a 
difference. Here too, the quality of their training and TA skills is at issue. 

An Unstable Target Audience: The Western Kenya region of Africa has a 

particularly high incidence of AIDS and Kisumu is no exception. In many of our visits to 
schools, we encountered staff making collections to help pay for funerals, teachers out on 
long-term leave due to illness, or absent so they could attend a funeral. The former Project 
Director reported a high number of AIDS-related teacher deaths in project schools. In 
addition, teacher and head teacher transfers were numerous and frequent. These two factors 
combined suggest that SIP-trained teachers and head teachers may not remain in the Kisumu 
schools for one reason or another and that large numbers of new staff will need to be trained. 
If replacements are not trained to a high level of expertise, then the impact of SIP will 
dissipate rapidly. Moreover, the number of schools in the Municipality has tripled in the past 
year, adding approximately 100 new, small, rural schools to the MEO s responsibility. If SEP 
POs are expected to address the needs of these schools also, particularly with limited 
transportation and new staff, their impact is highly questionable. 



In the next section, the views of teachers, head teachers, and school committees 
regarding the likelihood of sustainability are summarized. 
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Teachers’ and Head Teachers’ Views of Sustainability: Teachers were asked how 
confident they -were that the quality of training and support provided by SEP will continue after 
AKF support is removed and were asked to explain their answer. Their responses indicated an 
optimism derived from the achievements and motivation already engendered by SEP, but there 
seemed to be great concern about the management abilities of the MEO and whether materials 
would continue to be provided to teachers and classrooms once external funding ceases. 
Below is a sample of their responses: 

Those that expressed confidence cited the following reasons. Note that most of their 
comments are conditional . . . if. . . then. . .: 



■ If the whole SEP team goes to the MEO’s office and they continue in their current 
roles, there will be no change - SEP will continue. 

■ The training and TA already provided have established a sound base. 

■ The motivation is there. 

■ If the POs provide support to the inspectors and head teachers, there is no question of 
it not succeeding. 

■ If the funds are available to provide support to schools. 

■ If the TACs can assist us in making teaching aids. 



Those that expressed doubt cited the following concerns: 

■ It will all depend on the management by the MEO. 

■ Most of us are not equipped yet and we still need them (SEP POs). 

■ The POs may not visit schools regularly. 

■ I haven’t seen the MEO staff handle this type of work. 

■ The politics within the MEO may undermine SIP’s effectiveness. 

■ There are new inspectors who need in-service training. They don’t understand the 
types of instructional strategies that SEP has promoted. 

. Parents might not be able to buy the materials. Teachers will always work hard when 

materials are available. 

Related to the issue of sustainability, teachers and head teachers were asked if they felt 
it is important to maintain the role of the SIP Project Officer. 

■ What they are doing is what the TAC tutors should be doing. They need to reconcile 

their roles with the other MEO staff. . 

. They demonstrate difficult areas of the curriculum and assist in providing materials. 

■ They need to be here to help other schools. 

. Sometimes if you request a lesson, they’ll teach a lesson while you observe. 
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■ Because most schools in the Municipality have not implemented what was supposed 
to be implemented, SIP has not taken off properly. 

■ They’re important to help improve standards of teaching. 

■ Without them, the teachers would relax - they won’t sustain what they learned in the 
workshops; the SIP POs encourage teachers. 

Summary: 

Overall, the likelihood that SIP’s impact will be sustained over time is highly 
questionable due to a variety of factors including high rates of turnover in school staff and 
POs, limited transportation, currently untrained POs, triple the size of the target audience, 
overriding and non-child-centred influence of the KCPE, and quality dissipation. 
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CHAPTER 4 

DISCUSSION AND RECOMMENDATIONS 



Discussion 

SIP was quite successful in engendering broad-based support for what it was trying to 
accomplish. Attitudes toward the project are positive and widespread. Parents, teachers, head 
teachers, TAC tutors, inspectors and MEO administrators all expressed glowing goodwill 
toward the project’s goals and its staff. SIP has been exceptionally successful in getting all 
facets of the Kisumu education system to rally round SIP and to want to try to make it work. 
Everyone we spoke with knew about SIP and, almost to a person, were enthusiastic. The 
enthusiasm seemed most attributable to the materials and secondarily to the co-operative and 
helpful spirit of the SIP staff. 

SIP staff tackled all levels of the system. They worked closely with the MEO’s senior 
managers; offered training and technical assistance to the front-line MEO staff, inspectors and 
TAC tutors; encouraged the MEO to upgrade and expand the TACs; rewrote inspection 
guidelines to be consistent with SIP objectives; motivated head teachers to get parents to 
support the building of latrines, and the addition of doors, wire mesh screens, locks, desks, 
and chairs. Two head teachers we visited in schools about to enter SIP could hardly contain 
their excitement and insisted that we see the new desks that were being built. It is really hard 
to imagine how SIP could have been more successful in the area of public relations, 
commitment, and co-operation. 

However, we did not consider the teaching in the SIP classrooms to be particularly 
child-centred - the major focus of the SIP training. Teachers’ explanations of child-centred 
teaching and learning were generally shallow and often represented misconceptions of the 
notion. Although Phase 2 teachers who have been in the programme for several years 
exhibited significantly more child-centred teaching behaviours than did teachers in all other 
groups, few teachers actually had pupils work in small groups, engaged pupils in group 
discussions, or asked open-ended questions that would promote thinking, analysis or 
reasoning. SIP had little impact on the child-centred teaching behaviours of the upper primary 
teachers. Moreover, the regression analyses showed that child-centred teaching did not have a 
positive influence on test scores, although good classroom management did. This is probably 
because the child-centred strategies were not integrated to a level that they would make a 
significant difference. Perhaps more importantly, we saw very little evidence that SIP had 
made an impact on teachers’ skills in teaching reading or writing, despite having held several 
workshops in this area. 
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Comments made during the teacher interviews and the focus group sessions with 
school staff suggest that the curriculum and exams exert a more powerful influence on 
teachers’ behaviour than does the pedagogic appeal of child-centred teaching. “We don’t do 
what SIP wants because we need to cover the syllabus and we only have 35-minute periods.” 
“ If they cut back on the amount of material covered in the curriculum, we would have more 
time to engage the children in thinking, but now we avoid asking questions that might require 
any extra time. Some of the topics are too detailed for primary level and many topics are 
repeated in several subjects.” “ SIP methods require too much time.” Most teachers feel that 
the curriculum is too full and that they must press through all the topics in order to cover 
everything that the students will need to pass district, zonal, municipal, and national 
examinations. The success of their pupils is their highest priority, and success is judged by 
performance on the KCPE and by the interim regional and local exams. 



In addition, each class period is only 35 minutes, making it difficult to do activity- 
based lessons in such a short timeframe. SIP addressed these issues in several ways, they 
showed teachers and head teachers how to do a block schedule that would allow them 60-to- 
90-minute lesson segments. They taught teachers to identify and combine topics taught in 
several places in the curriculum. And they worked with teachers and other Municipal staff to 
develop a set of Municipal exams that would better reflect the local context. However, they 
may have actually undermined the child-centred teaching they were promoting by these 
exams. The exams were almost identical to the type administered by the KCPE and not 
supportive of child-centred teaching and learning. This is a critical leverage point that can be 
manipulated to create an environment more supportive of child-centred teaching and is 
discussed further in the recommendations. 

With regard to changes in student achievement, there are limitations in the conclusions 
that can be drawn from the findings due to the fact that pre-programme test data was not 
collected. Pre-post data would provide greater confidence in judgements about achievement 
gains attributable to SIP. However, the fact that there was such a clear distinction among the 
three levels of SIP intervention suggests that the impact of SIP is more likely to be felt over 
time. But, it is possible that SIP selected better schools to participate in the early stages of the 
programme and that poorer schools were selected later. In fact, there is some evidence of this. 
For one thing, the T3 schools had a much higher number of pupils absent from school on the 
day of the observation, 17 absences versus 6 and 4 for T1 and T2 respectively, and the average 
class size is much larger in the T3 classrooms we observed - 71 versus 49 and 45 for T1 and 
T2 and 54 for Siaya. 



102 



Evaluation of Phase 2 of the Kisumu School Improvement Project 



Capper, Nderitu, Ogula 



On the tests given for this study, when external factors were not controlled for, the 
schools that were involved in SIP during Phase 1 consistently performed significantly better 
than all other groups at the standard 3 level, and significantly better than the T3 and control 
schools at the standard 6 level. However, the control schools outperformed the T2 schools in 
two of the tests and performed about the same in another four tests — without the extra 
training and TA provided by SIP. The control schools outperformed the T3 schools in every 
test. 



When external factors such as family income and mother’s education were controlled 
for in the regression analyses, only T2 schools performed significantly better than the control 
schools at the standard 3 level in both the English and mathematics short answer tests, 
confirming that SIP did have a positive influence on pupil learning during Phase 2. SIP’s 
workshops also had a significant and positive impact on pupil learning. For each workshop a 
teacher attended, a pupil’s test score was likely to increase by over two points and teachers in 
the T1 group attended significantly more workshops than did any other group. This helps to 
explain the higher scores of pupils in T1 schools who, across all tests, achieved scores that 
were 16 percent greater than those of the control group, while pupils in T2 and T3 both 
achieved scores that were 5 percent greater than the control group. Although adjusted test 
scores at the standard 6 level were not significantly greater than scores of the control pupils, 
the SIP pupils at this level achieved a 13 percent increase in scores across all tests when 
compared with control pupils. The cost to increase the test scores of the standard 6 pupils was 
less than the costs for any of the standard 3 groups. 

Students in both treatment and control groups performed reasonably well on die short- 
answer and multiple-choice tests, suggesting that they are learning what is in the curriculum in 
the way that it is taught in the curriculum and textbooks. However, scores were exceptionally 
poor on all open-ended tests for all groups - around 10%. Many pupils wrote nothing at all or 
simply rewrote the prompt. There is clear evidence that they are not learning how to read, 
write, or communicate in English or Kiswahili. Nor are they learning to apply simple 
mathematical concepts and skills to real-life types of problems. 



Despite a recommendation in the Phase 1 evaluation report, SIP did not work with the 
whole school nor did they begin working directly with head teachers until late 1994. Since 
there is an abundance of research pointing to the importance of a strong instructional leader, it 
is likely that SEP could have had a substantially greater impact if it had engaged head teachers 
early on. SIP did, however, use School Co-ordinators to some extent for this purpose but their 
impact varied considerably from school to school. 
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Most of the head teachers we interviewed had no idea that they should be working 
with teachers to improve their instruction. They saw their roles as much more administrative, 
encouraging teachers and students to come to school and be on time, to collect fees and meet 
with parents, but not to observe or help improve teaching. The few head teachers who had 
attended SIP training in clinical supervision (observing and guiding teachers using a SIP- 
developed instrument aimed at child-centred teaching methods and classroom management) 
were enthusiastic about it and indicated that they planned to use it. However, their enthusiasm 
could easily slip away when confronted by the many other pressures of their jobs. In fact, 
most head teachers also teach several classes each day. This raises the question as to how 
much time they would actually have to observe and coach teachers. Perhaps sufficient time 
would be available if both the head teacher and School Co-ordinator were to work on 
instructional leadership tasks. Another possibility is to involve the MEO in trying to reduce 
the teaching load of the head teacher. 

In many of the schools we visited we found that teachers were absent, the school was 
closed, or some of the classes were away. Teacher absences were often due to illness or to 
attend funerals. But school closures and non-teaching days were generally due to music or 
sports festivals or to one of the many exams given by the various administrative levels of the 
system. These non-academic activities take place during the school day and are frequent and 
extended. For example, when we visited one of the newer SIP schools, the teacher was sitting 
at her desk relaxing while the children worked independently. When we told her that we 
would like to observe her teaching a lesson, she told us that she had not prepared because she 
was exhausted from the activities of the music festivals and that this was her first day at 

school in two weeks! 

There were about three days out of the three week data-collection period where some 
or all schools were closed due to events. Although national policy limits school closures for 
these purposes and requires that festivals and athletic events be held after school hours, the 
policies are ignored and not enforced by the MEO. These closures undermine the efforts of 
SIP and the aim of improved school quality and illustrate the importance of the MEO in 
articulating high academic standards and enforcing national policy that supports them. In fact, 
we found that one senior MEO official was unavailable for several days because he was 
attending athletic events in another district. 

Recommendations 



Toward the end of the data collection period, two members of the evaluation team met 
several times with the SIP Programme Officers to interview them regarding their views and 
experiences in implementing SIP. Their perspectives are particularly valuable in that almost 
all of the POs came directly from primary classrooms in Kisumu and so are able to view the 
effort from the perspective of SIP’s target audience. The evaluators asked the POs what they 
would do differently if they were designing a new school improvement programme - or what 
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recommendations they would offer to other communities designing a SIP-type project. Their 
suggestions were remarkably similar to the informal thoughts and discussions that the 
evaluation team members had as we visited schools, observed classrooms, and met with 
various representatives of the Kisumu school community. Although we have no structured 
data to support this contention, we all felt that the head teacher made the difference in the 
school. Where the head teachers were enthusiastic and highly engaged, school activities 
seemed to be more organized, efficient, and engaged. Where the head teacher was either 
uninterested or lax, the ethos of the school was also lax — teachers were absent, there was 
more milling about, facilities were in disrepair, etc. POs’ views were consistent with the 
extensive research documenting the importance of the head teacher as instructional leader. 

PO Recommendations: The following list contains the suggestions offered by the 
SIP Programme Officers and is followed by the evaluators’ comments and suggestions based 
on the study findings, informal observations, and recent research in school improvement, staff 
development and effective schools. 

■ Courses should be practical, based in schools, with follow-up provided immediately. 
Workshops are currently offered in central locations and there is often considerable time 
between the workshop and follow-up. 

■ All staff in a school should be trained together, including head teachers. They felt that 
school based training and assistance would result in a more sustainable intervention. Train 
TAC tutors, inspectors, and head teachers before training teachers so that the management 
of schools is supportive of, and consistent with, the skills and practices being promoted by 
SIP. They believe that it was a serious mistake to not train head teachers early on in the 
project and feel that SIP has had little impact in schools where the head teacher resisted 

SIP. 

■ Offer a one-month intensive training session for POs before they begin working with 

teachers. The training should include the following topics: project management, 

interpersonal skills, negotiating skills, school administration, school record keeping, 
school management skills, communication skills, and other topics based on the in-service 
needs of the teachers they will serve. They suggested that the training be sequenced so 
that they receive training in one or two topics, practice them in the schools or as they 
conduct workshops, receive feedback and coaching, and then move on to the next topic. 

■ Schools should be serviced in order of their needs. Those that have the most dire need 
should be attended to before those that have fewer and less significant needs. The level of 
intervention should vary based on a school’s needs. 

■ The staff should be involved in analyzing their work from a cost-effectiveness perspective. 
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■ The project should be evaluated every 18 months and changes made based on those 
evaluations. 

■ Reduce the number of schools to be serviced. Phase 2 specified that the project work with 
42 schools in a three-year period. Both the project staff and the Project Director reported 
feeling stressed with having to meet the “numbers”, when they often felt they were 
moving out of schools before the schools had fully integrated the new philosophies and 
practices. 

■ Dealing with fewer schools at a time would allow the staff more time to work together and 
to reflect on their work and make changes where needed. They reported that they often 
worked in isolation from each other because they were so busy. 

■ Begin by training one person at each school, such as the school co-ordinator, who could 
provide intensive and ongoing support and guidance to teachers. There was some debate 
about whether the school-based person should be the head teacher or a school co- 
ordinator. This person’s services would be backed up by the POs and TAC tutors. 
Intermittently, the Pos would provide intensive assistance to a school staff, perhaps 
staying with that school for a full week. There is some question as to whether head 
teachers would have the time to be instructional leaders even if they had the time and 
inclination to do so. 

■ Address the physical conditions and materials deficiencies of the school before beginning 
teacher training. 

Evaluators’ Recommendations: The following are categorized into training, 

programmatic, and policy recommendations. 

Training 

Study quality dissipation and improve training of project staff: We believe that a 
major reason for SIP’s limited impact is because the project staff were not trained to a 
sufficient level of mastery to do their jobs well. This is not intended as an indictment of the 
Programme Officers. They were dedicated, very smart, hard working and eager to leam. 
However, this is a problem that has plagued many similar projects and we believe is a problem 
that has not been faced head on in the development world. Very little is known about how 
much of the original training intent and quality is maintained as training proceeds down the 
line from the “expert” to other trainers and end-users. SIP POs are central to this model of 
teacher improvement and they are trained by experts; they then train others (such as the 
School Co-ordinators), who sometimes train others (such as teachers). How much of the lack 
of child-centred teaching was due to inadequate training and quality control of the training? 
There is evidence to suggest that the POs were not trained to mastery. They did not 
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demonstrate mastery during our observations of their workshops or classroom coaching. The 
POs reported that they did not receive training in coaching or consulting skills, and that they 
did not receive in-depth coaching as they worked to develop their own skills and knowledge. 
Nor did they have opportunities to read and discuss relevant research — other than in the 
degree programme, or have structured time to reflect on their craft with each other. And, the 
degree programme, while perhaps intellectually invigorating, was not practically oriented to 
the tasks and responsibilities of the POs. 

Another related issue is the length of time it takes to achieve a sufficient level of 
mastery for something to take hold. SIP POs reported that the teachers often had not 
solidified their understanding of the strategies before they (the POs) had to move on to another 
school. If a teacher does not quite understand how something is done, it is unlikely that he or 
she will incorporate the new skills into his or her teaching repertoire. 

Quality control is critical to the success of the SEP model. To ensure that the training 
and coaching delivered to teachers is accurate and effective, the POs need detailed, 
comprehensive training-to-mastery in a range of skills and knowledge, including child- 
centred teaching as applied to core subject areas, results-driven education, systems thinking, 
instructional leadership, group facilitation, planning, consulting, classroom-based coaching, 
student assessment, multiple forms of staff development, and strategies for promoting teacher 
networking. 

Two recommendations are offered. First, a small scale, ethnographic-type study 
should be conducted across several sites to assess the dissipation of training quality as it 
moves down through the levels, and to determine what it takes to retain quality. Second, a 
detailed curriculum and assessment system should be developed for those who will work as 
staff developers and their competence determined before they are allowed to assume full 
responsibility for their jobs. A greater degree of quality control may be built into this training 
if it is conducted through a central source, which leads to our next recommendation. 

Consider establishing a centralized training academy for staff developers, district 
educational managers, and head teachers: Although one option is to address the training of 
these role groups at the local level, another is to establish a centralized training academy. The 
need for training in these areas is broad and the curriculum needed rather extensive. It is not 
clear that a local project could either have sufficient quality control or be cost effective. A 
centralized academy could ensure both. Individuals who successfully complete the training 
for staff developers could then be assigned to local district offices to carry on the training and 
support efforts at the local level. Perhaps a requirement for participation would be that the 
district or municipality would guarantee that the individuals trained would assume appropriate 
roles of responsibility within a co-ordinated system when they complete training — and remain 
in those roles for a specified period of time. 
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With regard to head teachers and other district administrative staff, a better selection 
process and criteria may be needed. It appears that current selection is not always based on 
qualities related to performing the job well. Perhaps as a requirement for sending staff to the 
training academy, a district or municipality may be required to demonstrate that they have a 
defensible procedure and criteria for selecting individuals to serve as head teachers or 
inspectors — individuals who have the commitment and temperament for instructional 
leadership and community engagement. They might also be required to indicate how they 
plan to ensure a gender balance in head teachers. All but one of the head teachers in the study 
schools were males, while most of the teachers were females. 

A training academy could service several projects or even an entire country or region. 
After having proven its effectiveness in producing high-quality staff (head teachers, staff 
developers, district administrators, TAC tutors, etc.), it may be able to charge for these 
training and development services with the intention that eventually it could become 
financially self-sufficient. 

Produce materials centrally: One way to make the impact of such an academy more 
cost effective is to use centrally produced materials. Since AKF works in similar projects 
across many sites and countries, centrally produced training materials could be developed for 
district-based audiences, such as teachers and parents, as well as serving as supports for those 
who attended academy training. In this way, AKF would save the cost of having each project 
repeatedly devote time to creating such materials and could ensure high-quality, meticulously 
clear products based on the latest research in teaching, learning and school leadership. This 
would not preclude projects from developing additional materials to suit local needs. 



One form of training materials that has been proven to retain quality, induce learner 
engagement, and be cost effective when used with a large number of users is interactive video- 
based training materials, e.g., CD-ROM, digital video, etc. Interactive multimedia offers 
several advantages over traditional training models, the most important being that once the 
quality is built into the programme, it is there each and every time it is used. Once the 
programme is developed, multiple copies of the instructional software can be made very 
cheaply and distributed anywhere in the world, only requiring refilming and retaping for 
different ethnic and language groups. The essence of the instructional programme would 
likely remain the same for a role group across sites. 
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Other advantages include: learners can receive training when and where they need it;, 
the user is an active learner, able to control the pace and sequence of the material, and can 
review information repeatedly; the programme can be viewed individually, in small groups 
with or without a trained leader, or in large groups, and can serve as the basis for discussion 
and analysis among the participants; the same multimedia equipment can be used to support a 
variety of software training programmes; the user can view and interact with realistic imagery 
to assist in comprehending difficult concepts and to see how a teaching strategy actually 
works in a classroom setting. 

Regardless of how well POs are trained, they cannot master all the subjects, skills and 
topics for which school staff need training. If POs can be trained to be effective school 
improvement facilitators, the multimedia-based training can provide the high-quality content 
and skill based training needed across many sites and subjects. 

Promoting craft study -- increasing access to research literature and exemplary 
materials: Recent research on staff development reveals that it is important for educators to 
be actively engaged in studying their craft, rather than be more passive recipients of training. 
In order to study one’s craft, it is necessary to read relevant professional literature. Although 
SIP staff and other educators in the Municipality had some access to research literature 
through SIP’s Professional Development Resource Centre, the materials in the Centre were 
limited and often out of date. The research materials that the evaluators brought with them 
were borrowed and copied repeatedly. The staff were hungry for more and we believe it 
would be useful for them to have access to more current and regular publications and 
exemplary materials, such as those produced by the International Reading Association, or 
journals on teacher training. It is difficult to visualize effective strategies when one has 
seldom seen or read about them. The concepts are vague. Reading the research in an area and 
seeing how others have addressed it in the form of exemplary materials provides mental 
images that trainers and teachers can draw upon to guide their professional development. We 
suggest that the training of staff at all levels (POs, inspectors, TAC tutors, head teachers, and 
teachers) include reading either original research articles, or popularized research summaries 
and descriptions or copies of exemplary materials, discussing these in various group settings, 
talking about what the findings mean for their settings, how they might try it out, trying it out, 
observing and coaching each other, and meeting again several times to refine their thinking 
and use of what they have read. This, coupled with the video-based training materials 
described above promotes a craft-study approach to professional development and is more 
likely to infuse a deeper level of understanding and use. 

Programme 

Address school rather than teacher: We agree completely with the PO’s and the 
Phase 1 evaluation that the focus of SIP efforts should be on the school rather than on 
individual teachers within the school. School staff should be nurtured to work as a team to 
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identify and solve problems and to renew itself. The head teachers, POs, TAC tutors, and 
inspectors should be trained to assist the staff to pursue incremental annual improvement 
related to a set of common objectives, such as helping all students to become fluent and 
engaged readers. Rather than focus primarily on child-centred teaching strategies, child- 
centred teaching should be promoted within a broader framework of continuous school 
improvement and developing teachers’ subject expertise. 

Develop the head teacher as instructional leader: In order for a school to pursue 
incremental improvement, the head teacher must be able to create and sustain an academic 
focus and an ethos of improvement. He or she also must be able to motivate the community 
to provide resources and other forms of support. SIP has recently begun to address the role of 
the head teacher as instructional leader and these efforts should continue, perhaps even 
reducing the focus on teacher training in lieu of head teacher training. 

In order for head teachers to be effective instructional leaders, they will need training 
in the strategies of clinical supervision (observing, evaluating, and providing feedback to 
teachers), and in effective instructional strategies within specific subjects, especially reading, 
writing, math and science. It is also important for them to understand the notion of school 
improvement and how others have approached it successfully. If SIP POs and head teachers 
are trained at a central academy, then the head teacher is equipped to serve as an instructional 
leader and the PO is able to provide the external support needed to facilitate schoolwide 
improvement. 

Facilitate teacher networking: Teacher networking has been found to have a much 
greater impact on teacher practice than traditional staff development (Adams, 1992). Teacher 
networks involve teachers meeting with other teachers on a regular basis to develop 
curriculum materials, share experiences and frustrations, and further their understanding of an 
innovation. The most effective approach also involved a common within-school preparation 
period for teachers, cross-school monthly meetings, monthly staff development workshops 
that linked the treatment teachers with non treatment teachers, and a “ linker — a professional 
whose responsibilities included connecting teachers to an extended body of professional 
expertise through links with professional organizations and resources beyond the teachers’ 

district. 



One way to support teacher networking is to nurture subject specialists within a 
school. These might be teachers who are more skilled and knowledgeable in a particular 
subject area and could be provided with the additional training in the subject and pedagogical 
skills associated with that subject. These specialists could then be responsible for either 
teaching all of the classes in that subject, or providing assistance to other teachers. SIP cou ^ 
contribute to nurturing professional subject associations, such as the math teachers’ 
association, providing them with exemplary materials and research summaries of teaching and 
learning in their subject areas. They could initially facilitate monthly meetings of these 
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associations with the aim that these groups can assume a greater responsibility in the provision 
of the subject-based development of, and networking for, teachers. 

Emphasize the teaching of reading and writing and questioning strategies: 
Clearly, there are serious deficiencies in the teaching of reading and writing in Kisumu (and in 
the control district). In part these deficiences derive from their lack of emphasis in the 
curriculum, which focuses more on teaching decontextualized grammar than using language to 
understand and communicate. SIP has begun work on teaching reading, but much more needs 
to be done and should be a top priority in SIP’s future efforts. Current models of training 
teachers in reading and writing is to have them experience being readers and writers. This 
experience of the process helps them to understand the intellectual engagement and challenges 
their pupils will face and encourages them to be model readers and writers for their pupils. 
The International Reading Association has an international volunteer service to assist 
developing countries in this area and can be a valuable and cost-free asset. Moreover, several 
teachers asked for training in how to ask higher-level thinking-type questions. This would be 
very useful, especially since almost no thinking-type questions were asked during our 
observations. 

Develop, administer and report child-centred examinations: As mentioned 

repeatedly, the examinations exert a powerful hold on teaching and learning. SIP can capture 
the power of the exams and redesign them to nurture child-centred teaching. Instead of 
developing practice exams that promote superficial, fact-based teaching, they can develop 
exams that measure higher levels of thinking and problem solving, reading and writing, and 
the application of knowledge and skills to real-life problems and situations. Central to using 
examinations as a point of leverage is the timely reporting of exam results in a way that points 
out weaknesses and suggests teaching strategies that will lead to better exam performance in 
the future and are child-centred. That was the main purpose of including the open-ended 
questions in the exams administered for this evaluation and for developing detailed marking 
schemes. The questions measured important skills not measured in the “ traditional” exams - 
reading and writing - communicating ideas and concepts. They were intended to connect with 
pupils’ lives, feelings, and experiences and to elicit pupils’ views. They were intended to 
elicit imagination and creativity, and were marked for important writing skills, such as 
coherence. SIP (or the MEO) could provide teachers with sample test preparation exercises 
that embody these ideas. If the exam results are reported for each teacher, based on the ideas 
represented in these marking schemes, teachers will teach to the exams - in this instance, a 
desirable end. The inevitable question is whether these types of practice exams will 
undermine pupils’ performance in the national exams. Possibly, but not likely. The Kisumu 
pupils were not able to read and write in English - the language of the examinations. 
Certainly that is a major hindrance to high performance. In our view, there are no more 
important skills to achievement across all subjects than reading, writing (communicating), 
thinking, and problem solving. It is a proposition worth trying, and, if results are positive, 
could even encourage changes in the national examinations. 
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If this strategy is tried, Kisumu staff would need training in developing these types of 
exams and in reporting the results in ways that encourage better teaching. In addition, since it 
is a new strategy in Africa, the process and impact should be studied and reported broadly. 

Establish a model school: Teachers may not believe that they can really complete 
the curriculum while doing child-centred teaching. It may be useful to take one school 
preferably a school that is not privileged in any way - and work with the head teacher and staff 
until they have mastered school quality, child-centred teaching, and also have high test scores. 
This school could be used as a model for other schools to see that it truly can be done. 

Establish an effective schools peer review process: Research-based indicators of 
effective schools have been identified and could be used to establish an evaluation and 
improvement system within the municipality. At the present time, it is not clear that school 
staff are consciously aware of the factors that make up an effective school. School and 
municipality staff can work together to decide which indicators are appropriate for their local 
context and jointly design a procedure whereby a school is reviewed periodically - say once 
every year or year-and-one-half. When a school knows it will be reviewed in several months, 
and knows the criteria being used in the review, the staff may be motivated to analyze their 
own situation and work to make targeted improvements. Recall one teacher’s comment that, 
“Just the anticipation of SIP visits makes teachers work harder.” Based on the school review 
findings, the school staff and peer evaluation team could jointly establish specific goals for 
improvement for the next evaluation period. Ideally, staff from other schools and parents and 
community members, as well as faculty from the universities and teacher training colleges 
would all serve on review teams. This would require that staff at the MEO be trained to train 
the peer review teams — an appropriate task for AKF-SIP. 

Encourage MEO Responsibility: It is clear that SEP cannot assume full 

responsibility for the success or failure of schools in Kisumu. The MEO must exert a greater 
influence over school accountability. The MEO is responsible for ensuring that schools are in 
session when they should be in session and that festivals are attended after school hours. They 
should be more vigilant in checking on attendance and should work with SIP and other 
administrative levels to streamline the burden of multiple examinations. 

Policy 

Study of teacher training colleges: Teacher training colleges are a “black box” in 
the system of educational inputs. Little is known about their curriculum, the quality of 
teaching or supervision, or the adequacy of their facilities. A study of these institutions and 
the role they play in preparing teachers and head teachers may be revealing and may suggest 
strategies for improving a key part of the educational system in Kenya. 
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One of the POs who had been on the faculty of a teacher training college prior to 
coming to SIP, reported that he had previously felt that it was sufficient to present teachers 
with the theory behind the practice, but since his experience in working with teachers in the 
classroom, he sees the importance of providing teachers with a clinical, or practical, 
dimension to their teacher training experience. Perhaps this is an explanation for the gap 
between what teachers learn in teacher training and what they practice in the classroom. 
However, we suspect the problem may be broader. In our discussions with the faculty at 
Maseno University, we probed about the nature of the education curriculum and got the sense 
that it may be very much outdated. Since every teacher in Kenya goes through the teacher 
training colleges, helping to ensure that these institutions represent the best possible training 
may be a very cost-effective strategy for school improvement. Although most studies of pre- 
service training have found it to be less cost-effective than in-service training, this may be due 
to poor quality of the pre-service programs rather than to the inherent nature of pre-service 

training. 

Attempt to influence the design of the KCPE and the national curriculum: 
Although the KCPE is a quality test in many regards, especially in that the KNEC prepares 
and disseminates detailed reports of exam results with suggestions for instructional strategies, 
the examination is not at all child-centred. This is particularly true with regard to the 
extensiveness of the topics and subjects covered by the exam, which in part is a reflection of 
the controversially overburdened curriculum. One of the overarching principles of 
constructivism is the notion of meaningful learning, that is, learning which has meaning to the 
child. Numerous studies have shown that much of the learning that occurs in schools is 
superficial learning. Superficial learning is prompted by teachers feeling a need to cover an 
overly full curriculum. They press on even when they know that not all children understand 
the lesson Meaningful learning requires deep-level understanding, which in turn, requires 
that sufficient time be devoted to a topic. It also requires that learners be actively involved in 
constructing the knowledge. None of this is new to AKF - it is the reason for the focus on 
child-centredness in its school improvement projects. But the concepts are not reflected in the 
Kenyan curriculum or examinations. AKF’s school improvement projects will continue to 
encounter this obstacle of an overly full curriculum and national examination, but it is not at 
all clear how to overcome this obstacle. 

Collect baseline data: All new AKF projects should collect comprehensive baseline 
data at the beginning of their projects for use in subsequent evaluation studies. 
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